potiuk commented on issue #7947: Fixes too high parallelism in CI
URL: https://github.com/apache/airflow/pull/7947#issuecomment-605416262
 
 
   Hello @ashb @Fokko @BasPH @kaxil @turbaszek @mik-laj @feluelle - I think we 
need to merge i quickly.  I believe I found and fixed the problem that caused 
the last two days constant failures of our CI jobs. 
   
   It is very likely due to less resources (memory) available for the CI 
workers on Travis and I believe it was caused by too high default parallelism. 
In the tests we run we had default parallelism set to 32.  This means that for 
local executor, 32 worker processes started. I think we simply did not have 
enough memory to run all the processes and some workers crashed with 
out-of-memory exceptions and they caused the Local executor to get the EOF 
error on the queue reading from those processes. 
   
   I fixed it by detecting on CI how many processes we have (2 on Travis) and 
setting parallelism to that value.
   
   I got consistently failing tests without it and it started to work after 
that. There is one problem I have to deal with now - there is another 
"concurrent" test that is failing with those settings so I think I will settle 
on 4 rather than NUM_PROC originally. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to