dabla commented on PR #57975: URL: https://github.com/apache/airflow/pull/57975#issuecomment-3501051840
Don't know if this is a good idea when LazyXComIterator is consumed from a partial/expand, as each worker will prefetch to much data probably. Or I have to solve it in my code and decorate the LazyXComIterator with the solution above, or it has to be able to detect if it's called from a partial/expand, which I think is too complicated to achieve. I'll let this open at the moment to see what your guys thought's are, because this optimisation is for a specific case, namely one worker consuming the iterator in multi-threaded environment, which is different scenario than the partial/expand, as there the iterator will be consumed concurrently by different workers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
