1fanwang commented on issue #70953: URL: https://github.com/apache/airflow/issues/70953#issuecomment-5186672265
Verified [Drop unused post_submit_commands copy from SparkSubmitOperator](https://github.com/apache/airflow/pull/70323) on `apache-spark 6.3.1rc1`. Works as intended. Airflow 3.3.0, two identical virtualenvs: one with the RC providers, one with the previous stable release. Same Dag, same command, run through the real task runner. The external transport is stubbed, and the stub asserts the value that actually reaches it. Dead-attribute removal, so the check is that post-submit commands still run and the private copy is gone. Run at `LOGLEVEL=DEBUG` because the hook logs post-submit output at debug level. ```bash # apache-spark 6.3.1rc1 LOGLEVEL=DEBUG AIRFLOW_HOME=$PWD/airflow_home \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_70323_spark_submit \ .venv/bin/airflow dags test rc_70323_spark_submit # apache-spark 6.3.0, identical Dag LOGLEVEL=DEBUG AIRFLOW_HOME=$PWD/airflow_home_prev \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_70323_spark_submit \ .venv-prev/bin/airflow dags test rc_70323_spark_submit ``` ``` # apache-spark 6.3.1rc1 OPERATOR_HAS_DEAD_PRIVATE_COPY=False [debug] Running post-submit command: echo post-submit-ran-with-rendered-value [...SparkSubmitHook] loc=spark_submit.py:739 [debug] Post-submit command output: post-submit-ran-with-rendered-value state=success # apache-spark 6.3.0 OPERATOR_HAS_DEAD_PRIVATE_COPY=True [debug] Running post-submit command: echo post-submit-ran-with-rendered-value [...SparkSubmitHook] loc=spark_submit.py:736 [debug] Post-submit command output: post-submit-ran-with-rendered-value state=success ``` Identical post-submit behaviour on both, which is the intended outcome for removing something that was never read. The only delta is that the dead private copy is gone. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
