ecodina commented on issue #28647: URL: https://github.com/apache/airflow/issues/28647#issuecomment-1368017243
Hi @pankajastro We use SLURM to distribute tasks in different nodes of our cluster. The Airflow operator we use submits a job to SLURM. The workflow is as follows: Operator: log a few things & submit job & defer itself -> Trigger: check the SLURM log. When a few lines have been added, yield a TriggerEvent -> Operator: receive the new SLURM log lines in the TriggerEvent and log them in Airflow & defer itself again Therefore, all the logging happens inside the Operator, and not in the trigger. It looks like a frontend bug: when the operator is deferred (the trigger "runs point"), the website loses count of what try is happening and defaults to the last one. On Monday I'll try to write a simpler operator+triggerer (same structure but without using SLURM) to see if this keeps happening. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
