potiuk commented on pull request #13730: URL: https://github.com/apache/airflow/pull/13730#issuecomment-773678053
@ashb -> just to reiterate - those are very same failures I used to observe in many CI environments earlier. The biggest benefit of having full control over the CI machines is the ability to switch to in-memory-only operations. Switching to `tmpfs` solves the need of solving those kind of I/O failures in the cloud environment as well ups speed up everything several times (sometimes even 10 times). I spent hourse days on solving these kind of problems in the past and they are gone entirely with several times overall build time improvement when we switch to tmpfs for both the source code location (for any kind of node setup/precompiling/parsing/execution) and docker files (for actual image building/container execution etc.). I could have saved days of work last time i've done similar workarouds for spurious I/O failures - and then I stopped worrying about those kind of problems altogether when switched to tmpfs. This is like being in another world entirely. Just saying. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
