pegasas commented on PR #33117: URL: https://github.com/apache/airflow/pull/33117#issuecomment-1666515289
> Your proposals were (and continue to be) confusing. After those iterations of your proposals, I am still not sure what you are proposing and whether your read the code, because you make some assumptions without checking them and without stating that you are making assumptions. I get your point. I will refer to other AIP/issues to see what makes a suggestion clear. I tried to solve in this next proposal. > Similarly the issue is about being able to run multiple dag file processors on several different machines, parsing the same shared folder in order to scale horizontally. These solved my doubt. I will dig into this and tries to come to a clear suggestion. > The code you are referring to is wrong. What you are referring to is a generic DAGBag code. But you should look at what DagFileProcessor does. It spawns forks for every file it finds and the forked process got the FILE as input, not the folder. You missed the fact that DAGBAg can be given a file not only a folder. you are right. next time I will refer the right place to make my suggestion clear. make my proposal more focus. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
