pegasas commented on PR #33117: URL: https://github.com/apache/airflow/pull/33117#issuecomment-1665928097
Thanks @potiuk for clarification! Actually, I have considered this problem, too, do we have to do multi-process on a single file? cause as I see, each DagFileProcessorProcess read DAG from the file/zip and saves it to DB. https://github.com/apache/airflow/blob/af0839200565dd31787632119aec0b6a751b0ca8/airflow/models/dagbag.py#L317 If we make it multi-process and use synchronising between themselves, we have to increase their lock expropriation, but I think the bottleneck may be the File IO. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
