potiuk commented on PR #33117: URL: https://github.com/apache/airflow/pull/33117#issuecomment-1666487447
It's not about making people uncomfortable, but about trying to make your proposal clear so that people do not need to loose time on trying to understand what you want. All this should be backed not on assumptions but on looking at the code and if you do base it on assumptions, clearly state it. Your proposals were (and continue to be) confusing. After those iterations of your proposals, I am still not sure what you are proposing and whether your red the code, because you make some assumptions without checking them and without stating that you are making assumptions - from your coments it looks lke you know what you are talking about, but in fact you don't. It almost looks like you are using chatbot to generate your questions. They look vaguely familiar to those "hallucitanating" chatbots that seem to know what they are talking about but what they say is not based on facts. Check your assumptions, if you don't know - state your assumptions. And no. There is STILL somethig to do here. you might want to utilise not multi-processing but multi-node setup, similarly lile scheduler is horisontally scalable now. You can run mutlple schedulers and they will scale - the more nodes you have, the more CPU on multiple machines they will be able to use. Similarly the issue is about being able to run multiple dag file processors on several different machines, parsing the same shared folder in order to scale horizontally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
