amoghrajesh commented on issue #24171: URL: https://github.com/apache/airflow/issues/24171#issuecomment-4574438540
Same problem exists for spark on k8s. The `spark-submit` blocks for the full job duration watching the driver pod via `LoggingPodStatusWatcherImpl`, holding similar heap size while doing no actual work. Also blocks making the operator deferrable when/if needed. Planning to fix it with the same pattern as #65991 (yarn_track_via_rm_api): a `track_driver_via_k8s_api: bool = False` flag on `SparkSubmitHook/SparkSubmitOperator` that terminates `spark-submit` early after pod creation and polls via the K8s client instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
