pegasas commented on PR #33117: URL: https://github.com/apache/airflow/pull/33117#issuecomment-1666347005
Hi, @potiuk , Firstly I just want to understand what your proposal is.  > basically each file is processed in a new forked python process > BTW. Horizontal scaling of DAG file processor would really be needed if your Python Parsing gets a lot of CPU cycles. And mostly when you have big number of smal Python files. > The idea is to implement multiple Dag File Processors that work on the same --subdir and share the load - somehow synchronising between themselves so that they do not process the same files at the same time. **My Question Is: We already have one process (which is DagFileProcessorProcess) for each file for parallel processing, right? do you mean that we should fork more processes to process module, which is step 3 of DagFileProcessorProcess.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
