vardancse edited a comment on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#issuecomment-428557928 @ashb For us, we can easily reproduce by running a DAG which has a task bound to fail with retries set >0(preferably 4 or 5) and retry_delay set to 1 second, let me know if you're able to repro else I'll share sample DAG file. Also, our email_on_retry and email_on_failure is marked true, so once we see this issue of race condition, we will get an email something like below > Try 2 out of 4 > Exception: > Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally? > Log: Link > Host: 72a8e9235db3 > Log file: /usr/local/airflow/logs/1fac4a1d57e551320a49403c_ufr09_qwpbfy/ACPComputeGateway_0_UFR09Task/2018-10-03T05:45:05.000221.log > Mark success: Link > EDIT-1: We have also done 1 testing at our side, where we were not able to reproduce issue when dag_folder consist less number of files but as soon as it touched 40+ Dags in our dag_folder, we were able to reproduce, probably increasing dags in dag_folder increasing 1 cycle of dags processing.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
