davido912 opened a new issue #9564: URL: https://github.com/apache/airflow/issues/9564
Hey hey, Trying to adapt working using a dockerised version of Airflow (apache/airflow:1.10.10-python3.6) and load our repository to it. However, in comparison to using Airflow locally installed in a virtual environment, the dockerised version is extremely slow, operators queue for long as if there is only one worker that can take care of the jobs. And pipelines do not manage to execute in full due to the fact that the scheduler crashes - I still couldn't figure out why. The scheduler fires the following: Process QueuedLocalWorker-5: Process QueuedLocalWorker-7: Process QueuedLocalWorker-9: Process QueuedLocalWorker-16: Process QueuedLocalWorker-12: Process QueuedLocalWorker-17: Process QueuedLocalWorker-10: Process QueuedLocalWorker-6: Traceback (most recent call last): Process QueuedLocalWorker-14: ... ... ... SystemExit: 0 and eventually: BrokenPipeError: [Errno 32] Broken pipe. Did anyone experience this or anyone knows what is going on? Very much appreciate any feedback. Thanks!! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
