The migration is now complete and trees have been re-opened. Big thanks to Eric Maydeck for doing the hard bits and making this migration so smooth!
I reran any failures I noticed on try and they are all coming back green so far. If I missed any tasks, a rerun / retrigger should be all you need. Thanks for your patience! Andrew On Wed, Jul 12, 2023 at 2:03 PM Andrew Halberstadt <[email protected]> wrote: > Just a quick status update: > > The artifact storage backend has been cut over, but a final sync task is > still ongoing to sync the last delta of artifacts from S3 -> GCS. So if you > see any task failures around missing artifacts, this is expected. Once the > sync task wraps up in the next 30min or so, the artifacts should be > available again. At which point rerunning your tasks should make them green. > > On Wed, Jul 12, 2023 at 11:05 AM Andrew Halberstadt <[email protected]> > wrote: > >> Quick reminder that the planned tree closure is scheduled for 12:00 EST >> (one hour from now) and should last approximately two hours. Try will >> remain open. Updates will be posted to #firefox-ci >> <https://matrix.to/#/%23firefox-ci:mozilla.org> on Matrix and I'll >> follow-up here when the migration is finished as well. >> >> Please be on the lookout for suspicious failures, especially around: >> 1. Missing artifacts. >> 2. Failure to decompress artifacts. >> >> Please ping the #firefox-ci >> <https://matrix.to/#/%23firefox-ci:mozilla.org> channel on Matrix if you >> find any issues. >> Thanks, >> Andrew >> >> On Fri, Jun 23, 2023 at 2:43 PM Andrew Halberstadt <[email protected]> >> wrote: >> >>> *Update:* Firefox CI artifact migration and planned tree closure is >>> moved to July 12th @ 12:00 EST. >>> >>> The community migration was completed successfully, however a problem >>> related to how S3 and GCS handle gzip compressed files came up which will >>> likely cause some issues in Firefox CI (read below for details). Since June >>> 28th is preceding a long weekend followed by the Firefox 115 release, we've >>> decided to delay the migration until July 12th, when we'll be under no time >>> pressure to fix fallout. >>> >>> >>> >>> *Issue Details* >>> It turns out that S3 serves the file with whatever encoding it was >>> uploaded with no matter what. So if you request a gzip compressed artifact, >>> S3 will serve you that gzip compressed artifact. GCS on the other hand, >>> pays attention to the Accept-Encoding >>> <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding> >>> header. So if you request a gzip compressed artifact, GCS will serve you >>> the gzip compressed artifact if you specify `Accept-Encoding: gzip`, but >>> otherwise helpfully decompress the artifact (as the lack of the >>> `Accept-Encoding` header indicates you can't handle gzipped files). This >>> means that after the transition, places that were expecting downloaded >>> artifacts to be gzipped, may instead be decompressed. >>> >>> We'll use the extra time to perform extra testing and validation to try >>> and root out issues ahead of time. However, it is not possible to find >>> every place that downloads artifacts from Taskcluster. If you maintain any >>> tasks or tools that download gzipped Taskcluster artifacts and want to >>> avoid any disruption, please ensure you are passing the `Accept-Encoding: >>> gzip` header in your GET requests. >>> >>> - Andrew >>> >>> >>> >>> On Tue, Jun 20, 2023 at 12:32 PM Andrew Halberstadt <[email protected]> >>> wrote: >>> >>>> Hey everyone, >>>> >>>> We're planning a tree closure to migrate Firefox CI's artifact storage >>>> backend from Amazon S3 to Google Cloud Storage: >>>> >>>> *When:* June 28th @ 12:00 EST >>>> *Duration:* 2hrs (or longer if we need to rollback) >>>> *What to expect:* >>>> Try will remain open. While we anticipate the transition should be >>>> mostly seamless, you may notice some recent artifacts will be missing for a >>>> short period of time following the cutover (but they will be synced over >>>> soon) and some unlucky tasks may fail with an exception and need to be >>>> rerun. >>>> >>>> We'll also be migrating the Community Taskcluster instance tomorrow in >>>> order to detect any issues that arise beforehand. If you notice missing >>>> artifacts or other issues, please reach out on #firefox-ci >>>> <https://matrix.to/#/%23firefox-ci:mozilla.org> (for issues with the >>>> Firefox CI instance) or #taskcluster >>>> <https://matrix.to/#/%23taskcluster:mozilla.org> (for issues with the >>>> Community instance). >>>> >>>> Big thanks to Eric Maydeck for all the prerequisite work to make the >>>> migration possible! >>>> >>>> Please reach out if you have any questions or concerns. >>>> Cheers, >>>> Andrew >>>> >>> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAAJAz%2B52EZ0JfwynZaX7TKFBuw5OGWJcm2A0FUfCcso6%2Bsf_7A%40mail.gmail.com.
