[
https://issues.apache.org/jira/browse/AIRFLOW-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208658#comment-16208658
]
ASF subversion and git services commented on AIRFLOW-1631:
----------------------------------------------------------
Commit cb868f49f0b6144bd6488cbde5bdf2811001f6ac in incubator-airflow's branch
refs/heads/master from [~erod]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=cb868f4 ]
[AIRFLOW-1631] Fix timing issue in unit test
LocalWorker instances wait 1 sec for each unit of
work they perform, so
getting the response of a processor takes at least
1 sec after the unit
of work.
Increasing timeout in
LocalTaskJobTest.test_mark_success_no_kill and
decreasing the poking time on local executor
checking for results in the
unlimited implementation.
Closes #2699 from edgarRd/erod-fix-timing-failure
> LocalExecutor does not maintain contract of unbound parallelism (0 value)
> -------------------------------------------------------------------------
>
> Key: AIRFLOW-1631
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1631
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor
> Affects Versions: 1.8.1
> Reporter: Edgar Rodriguez
> Assignee: Edgar Rodriguez
>
> *Location*
> {{airflow/executors/local_executor.py:LocalExecutor#start}}:
> {code}
> def start(self):
> self.queue = multiprocessing.JoinableQueue()
> self.result_queue = multiprocessing.Queue()
> self.workers = [
> LocalWorker(self.queue, self.result_queue)
> for _ in range(self.parallelism)
> ]
> for w in self.workers:
> w.start()
> {code}
> *Description*
> When *{{PARALLELISM}}* configuration value is set to {{0}}, using local
> executor will stall computation since it won't create any workers. As
> described in base_executor:
> {code}
> :param parallelism: how many jobs should run at one time. Set to
> ``0`` for infinity
> :type parallelism: int
> {code}
> Hence, this contract is not maintained in {{LocalExecutor}}.
> *Remediation*
> In the context of local executor, in theory if parallelism is unbounded, then
> every task submitted should run immediately in its own process, so we could
> spawn a process for each submitted task without using a worker queue. When
> the task is completed, the worker can be terminated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
