potiuk commented on pull request #14531: URL: https://github.com/apache/airflow/pull/14531#issuecomment-790050472
> Not once in that comment you linked to did you say that you tested it on AWS. The whole comment was after the PR failed. I is exclusively about the tests that are failing in this PR and justifying the need why we need bigger than 32 - II even copy&pasted the output from the configuration printed by the failing tests and the machines in AWS. There. And I explained that it takes 26 minutes to fail (in those very tests which were originated by the PR). And THEN I explained why I believe 64 GB as needed (because corresponding tests were taking 35 GB on my machine). Here is the beginning of the comment. All of it refers to tests that failed on our self-hosted architecture on 32 GB machine. I re-read it again, and I think you cannot read it differently. I am just explaining why we need bigger machines for our tests (which are - obviously running on our AWS infrastructure). > Hey @ashb - we need bigger machines as I suspected :) . > The good news is that it will be much cheaper in the long run as we will need them for far less time. > The tests are failing but mainly because of memory problems and timeouts (so I guess we are simply using too much of RAM , if we up the machine to 64 GB I think this should go rather smoothly. The good news is that even with not enough memory (and with failures/timeouts) the tests took ~26 m (!) for sqlite - rather than > 1 h, so when we have enough memory we can achieve the 15 minutes I was hoping for. Those 64 GB machines are only a bit more expensive than the 32 GB ones, so we will save a lot of credits when it works. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
