[
https://issues.apache.org/jira/browse/AIRFLOW-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164391#comment-17164391
]
Kamil Bregula commented on AIRFLOW-4481:
----------------------------------------
In Airflow 1.10.10, dummy operators is evaluated by the scheduler but never
processed by the executor.
https://github.com/apache/airflow/pull/7880/files
> DummyOperator performance improvement - currently needs +87 seconds to do
> nothing
> ---------------------------------------------------------------------------------
>
> Key: AIRFLOW-4481
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4481
> Project: Apache Airflow
> Issue Type: Improvement
> Components: DAG, operators
> Affects Versions: 1.10.3
> Reporter: t oo
> Priority: Trivial
>
> I don't know why a DummyOperator can't finish in sub-second time, currently
> taking >4 seconds.
> *** Reading local file:
> /home/ec2-user/airflow/logs/stream_run/JOIN_tsk_/2019-01-01T00:00:00+00:00/1.log
> [2019-05-08 15:33:53,982] {__init__.py:1139} INFO - Dependencies all met for
> <TaskInstance: stream_run.JOIN_tsk_ 2019-01-01T00:00:00+00:00 [queued]>
> [2019-05-08 15:33:54,009] {__init__.py:1139} INFO - Dependencies all met for
> <TaskInstance: stream_run.JOIN_tsk_ 2019-01-01T00:00:00+00:00 [queued]>
> [2019-05-08 15:33:54,009] {__init__.py:1353} INFO -
> --------------------------------------------------------------------------------
> [2019-05-08 15:33:54,009] {__init__.py:1354} INFO - Starting attempt 1 of 1
> [2019-05-08 15:33:54,009] {__init__.py:1355} INFO -
> --------------------------------------------------------------------------------
> [2019-05-08 15:33:54,026] {__init__.py:1374} INFO - Executing
> <Task(DummyOperator): JOIN_tsk_> on 2019-01-01T00:00:00+00:00
> [2019-05-08 15:33:54,026] {base_task_runner.py:119} INFO - Running:
> [u'airflow', u'run', 'stream_run', 'JOIN_tsk_', '2019-01-01T00:00:00+00:00',
> u'--job_id', '2550', u'--raw', u'-sd',
> u'DAGS_FOLDER/stream_dag_generation.py', u'--cfg_path', '/tmp/tmpJR_rQX']
> [2019-05-08 15:33:56,117] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,117] {settings.py:182} INFO -
> settings.configure_orm(): Using pool settings. pool_size=5,
> pool_recycle=1800, pid=16385
> [2019-05-08 15:33:56,317] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,317] {__init__.py:51} INFO - Using executor
> LocalExecutor
> [2019-05-08 15:33:56,582] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,581] {__init__.py:305} INFO - Filling up the
> DagBag from /home/ec2-user/airflow/dags/stream_dag_generation.py
> [2019-05-08 15:33:58,002] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:58,001] {cli.py:517} INFO - Running
> <TaskInstance: stream_run.JOIN_tsk_ 2019-01-01T00:00:00+00:00 [running]> on
> host domain
> [2019-05-08 15:33:58,279] {logging_mixin.py:95} INFO - [2019-05-08
> 15:33:58,279] {jobs.py:2562} INFO - Task exited with return code 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
