[jira] [Commented] (AIRFLOW-1493) Fix race condition with airflow run
[ https://issues.apache.org/jira/browse/AIRFLOW-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156021#comment-16156021 ] ASF subversion and git services commented on AIRFLOW-1493: -- Commit b2e1753f5b74ad1b6e0889f7b784ce69623c95ce in incubator-airflow's branch refs/heads/master from [~saguziel] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b2e1753 ] [AIRFLOW-1493][AIRFLOW-][WIP] fixed dumb thing Closes #2505 from saguziel/aguziel-fix-double- trigger > Fix race condition with airflow run > --- > > Key: AIRFLOW-1493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1493 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > > Currently, airflow run spawns a process `airflow run --local` which spawns > `airflow run --raw`. > Local manages the heartbeat. Raw performs a series of checks, sets the state > to running, runs the task, then sets the state to failed or success. > The problem is the heartbeat check on `airflow run --local` has to monitor > the state in the DB, but because the change of state to running happens > asynchronously, it must first observe the state in the DB to be running > before it has the power of termination. However, there is no guarantee that > it will observe this state. Thus, we should move the pre-execution logic to > airflow run --local -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1493) Fix race condition with airflow run
[ https://issues.apache.org/jira/browse/AIRFLOW-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118305#comment-16118305 ] Bolke de Bruin commented on AIRFLOW-1493: - Why not rely on the OS to verify the task will actually run? I.e. Use signals, these are observable without the issue your are describing. > Fix race condition with airflow run > --- > > Key: AIRFLOW-1493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1493 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > > Currently, airflow run spawns a process `airflow run --local` which spawns > `airflow run --raw`. > Local manages the heartbeat. Raw performs a series of checks, sets the state > to running, runs the task, then sets the state to failed or success. > The problem is the heartbeat check on `airflow run --local` has to monitor > the state in the DB, but because the change of state to running happens > asynchronously, it must first observe the state in the DB to be running > before it has the power of termination. However, there is no guarantee that > it will observe this state. Thus, we should move the pre-execution logic to > airflow run --local -- This message was sent by Atlassian JIRA (v6.4.14#64029)
