vincbeck commented on PR #32572: URL: https://github.com/apache/airflow/pull/32572#issuecomment-1739769426
> I only came here cus I noticed the new warning as I was squashing all the warnings in my local dev setup after rebasing. Not a problem, always good to ask questions :) > And reading it again, what you are saying is, prior to your PR, when max_tis_per_query is greater than parallelism, it has no effect whatsoever. I.e. it only changes behavior when it is less than parallelism. And therefore, you warn the user when it's configured that way. This is not my PR but happy to give my opinion on this one :) > But that warning does not look to be quite accurate, so maybe we should just revert that warning. What do you think? > Here it seems the setting is used without reference to cluster parallelism. I am just wondering what is wrong: - Is the warning wrong because this is not entirely true because, as you said, parallelism is not taken into consideration here https://github.com/apache/airflow/blob/7a5b6a39c948bc51d754898b0478ad319dfab43f/airflow/jobs/scheduler_job_runner.py#L1491 - Should we take parallelism into consideration at the line above -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
