1fanwang commented on issue #70953: URL: https://github.com/apache/airflow/issues/70953#issuecomment-5186668101
Verified [Reconnect to the running Livy batch on retry](https://github.com/apache/airflow/pull/68956) on `apache-livy 4.6.0rc1`. Works as intended. Airflow 3.3.0, two identical virtualenvs: one with the RC providers, one with the previous stable release. Same Dag, same command, run through the real task runner. The external transport is stubbed, and the stub asserts the value that actually reaches it. A synchronous `LivyOperator` with `retries=1`. The stubbed hook raises during the first poll to simulate the worker dying mid-run, and counts how many batches were submitted across both attempts. ```bash # apache-livy 4.6.0rc1 AIRFLOW_HOME=$PWD/airflow_home \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_68956_livy_reconnect \ .venv/bin/airflow dags test rc_68956_livy_reconnect # apache-livy 4.5.7, identical Dag AIRFLOW_HOME=$PWD/airflow_home_prev \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_68956_livy_reconnect \ .venv-prev/bin/airflow dags test rc_68956_livy_reconnect ``` ``` # apache-livy 4.6.0rc1 SUBMITTED_BATCH_TO_CLUSTER call#1 -> batch_id=42 POLL#1 batch=42 -> simulating worker crash mid-poll [info] Task instance state updated ... new_state=up_for_retry [info] Reconnecting to existing job [...LivyOperator] external_id=42 external_id_key=livy_batch_id loc=resumablejobmixin.py:160 status=running TOTAL_BATCHES_SUBMITTED_TO_CLUSTER=1 state=success # apache-livy 4.5.7 SUBMITTED_BATCH_TO_CLUSTER call#1 -> batch_id=42 POLL#1 batch=42 -> simulating worker crash mid-poll SUBMITTED_BATCH_TO_CLUSTER call#2 -> batch_id=42 TOTAL_BATCHES_SUBMITTED_TO_CLUSTER=2 AssertionError: expected exactly ONE batch on the cluster, got 2 ``` The retry reconnects to the batch already running instead of leaving an orphan Spark job on the cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
