sbp commented on issue #816: URL: https://github.com/apache/tooling-trusted-releases/issues/816#issuecomment-4908026135
The release watcher has been running in the staging version of ATR for three weeks, so we can now get a count of release frequency. The raw count needs two corrections, however: a single Airflow providers commit on 22 June was reported as thirty separate releases because the batch collapsing logic had not yet been added, and the watcher only reports releases from projects that it can match in ATR's database. Taking these corrections into account, we saw about 50 non-ATR releases in three weeks, roughly 70 per month, and we also need to add ATR's own rate of about 13 per month to that. We therefore need to accommodate around 90 to 110 releases per month, five times our current alpha rate. Most projects released just a few artifacts. In the average release, ATR writes about six hundred check results and just over twenty task rows to the database, which comes to less than half a megabyte, and which uses only a few seconds of CPU, nearly all due to RAT. The biggest cost is disk. Measured across everything currently retained in the dist release area the median release is a few megabytes, but the mean is about 130 MB because a small number of projects ship enormous sets with many GBs of artifacts. When a release is being composed on ATR, its extracted archives occupy roughly two to four times the artifact size on top. Ordinary releases are nearly free, so provisioning is really a question of when the largest releases happen. OpenOffice, for example, makes 40 GB releases just in artifacts. Peak resource usage after we go to beta is also shaped by how many releases are being worked on simultaneously. Releases are worked on in ATR for about a month on average. If we count finished releases only, the average is two weeks, so unfinished releases are substantially driving up that average. At 100 releases per month we should expect about 100 simultaneous releases. At 100 releases per month, an ordinary day needs about 100 GB: 13 GB of unfinished files, 26 GB of extracted archives, up to 16 GB of extraction workspace, about 6 GB of database and rotated logs, and around 40 GB of Docker images and build cache. Then we have to factor in spikes. When the largest releases are made, that adds the requirement of tens of GBs for the archives, and double that again for extracted archives. OpenOffice's release is 40 GB of files, of which 27 GB are tarballs and zips. I didn't test what these extract to, but assume 30 to 80 GB, so one such release adds as much as 120 GB. Some large release projects such as kie and pulsar release frequently; we should assume that there will be about three such large releases per hundred. Peak disk usage would therefore be over 220 GB, and we should have a large margin on top. The task worker pool grows from 4 to 8 processes under load by itself, so 8 cores would be necessary to keep the web process happy, but we need to decide how many task workers to use. These numbers would change quite a bit if we change our current policies. Cleaning extracted archives at announce helps a lot, because that cache would otherwise grow by tens of gigabytes per month. We should consider pruning completed recurring status tasks, which make up nearly all of the largest database table, and rotating the logs. Measuring the memory usage of task workers required adding more logging to ATR, which I did in 7ad6ddd3909644b887221950517b09273b6ac2b4. We'll need the data from this to accurately assess how much RAM we need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
