zozo123 commented on PR #76279: URL: https://github.com/apache/airflow/pull/76279#issuecomment-6630097792
Thanks @Dev-iL, fair point about steal time, so I checked the logs as you suggested. I sampled Tests (AMD) runs from Sep 1 to 30 and Oct 2 to Nov 3 and read the annotations of every failed or cancelled job (639 jobs). Build CI image had 3 failures in 183 jobs before #73997 and 6 in 182 after. None was a lost communication error and none failed in the Export step. Four of the six after are one Import mount-cache incident on Oct 9 within 40 minutes ([example](https://github.com/apache/airflow/actions/runs/37989368691/job/114020050692)). Across all sampled jobs I found 3 lost communication failures after Oct 1 and 0 before. All 3 were test jobs where nothing is compressed ([Oct 5](https://github.com/apache/airflow/actions/runs/37368809147/job/111966728016), [Nov 3](https://github.com/apache/airflow/actions/runs/41712009533/job/126186355099)). The growth in failed minutes lines up with more test failures and with RunsOn spot interruptions, which started in the same window. On hosted runners, median export plus stash went from about 281s to 138s, and zstd -6 peaks at about 20 MiB per thread. A sample can't rule out a rare effect, so for a safety margin I'm happy to cap zstd at nproc minus 1 in both the CI and PROD builds. I don't see anything in the data that calls for a revert. cc @shahar1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
