sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle
error occurs when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#discussion_r365555136
##########
File path: tests/test_utils/mock_executor.py
##########
@@ -36,10 +36,17 @@ def __init__(self, do_update=True, *args, **kwargs):
self.history = []
# All the tasks, in a stable sort order
self.sorted_tasks = []
- self.mock_task_results = defaultdict(lambda: State.SUCCESS)
+
+ # If multiprocessing runs in spawn mode,
+ # arguments are to be pickled but lambda is not picclable.
+ # So we should pass self.success instead of lambda.
Review comment:
If we run `TestSchedulerJob.test_retry_still_in_executor` on macOS, we will
get following stack-trace.
```
File "/Users/kou/work/oss/airflow/airflow/jobs/scheduler_job.py", line
1499, in _execute
self._execute_helper()
File "/Users/kou/work/oss/airflow/airflow/jobs/scheduler_job.py", line
1532, in _execute_helper
self.processor_agent.start()
File "/Users/kou/work/oss/airflow/airflow/utils/dag_processing.py", line
353, in start
self._process.start()
File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/process.py", line
121, in start
self._popen = self._Popen(self)
File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/context.py", line
224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/context.py", line
283, in _Popen
return Popen(process_obj)
File
"/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line
32, in __init__
super().__init__(process_obj)
File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_fork.py",
line 19, in __init__
self._launch(process_obj)
File
"/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line
47, in _launch
reduction.dump(process_obj, fp)
File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/reduction.py", line
60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object
'MockExecutor.__init__.<locals>.<lambda>'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
