dwreeves commented on issue #67515: URL: https://github.com/apache/airflow/issues/67515#issuecomment-4550213511
Thanks for the reply and the explanation. Not that it really matters, but I think that "Worker running a non-Neo4j task in a DAG that mentions Neo4jOperator" is actually quite common? That was the main use case I was interested in. I have DAGs with, say, 12 tasks, one of which references an operator that brings along with it a lot of global imports. So there are no savings for that one task, but there are savings for the other 11. The other 11 may be very trivial and can run on queues with tiny workers. I have experienced random OOM issues (I'll admit, in Airflow 2) attempting to run things on those tiny workers. Fail-fast loss is a real regression, I agree. I was operating under the assumption that the global imports were required deps, but did not consider the case of an SDK version mismatch. PEP 810 being the right path forward makes sense. Thanks again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
