potiuk commented on code in PR #33144: URL: https://github.com/apache/airflow/pull/33144#discussion_r1286873162
########## dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md: ########## @@ -0,0 +1,328 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Purpose of the document](#purpose-of-the-document) +- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci) +- [Manually refreshing the image cache](#manually-refreshing-the-image-cache) + - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually) + - [Prerequisites](#prerequisites) + - [How to refresh the image cache](#how-to-refresh-the-image-cache) + - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache) + - [What the command does](#what-the-command-does) +- [Manually generating constraint files](#manually-generating-constraint-files) + - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually) + - [How to generate constraint files](#how-to-generate-constraint-files) + - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually) +- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files) + - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely) + - [How to update the constraints](#how-to-update-the-constraints) + - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually) + - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-) + - [Examples of running the command](#examples-of-running-the-command) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +# Purpose of the document + +This documents contains explanation of a few manual procedures we might use at certain times, to update +our CI and constraints manually when the automation of our CI is not enough. There are some edge cases +and events that might trigger the need of refreshing the information stored in our GitHub Repository. + +We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors +and users: + +* `CI and PROD image cache` - used by our CI jobs to speed up building of images while CI jobs are running +* `Constraints files` - used by both, CI jobs (to fix the versions of dependencies used by CI jobs in regular + PRs) and used by our users to reproducibly install released airflow versions. + +Normally, both are updated and refreshed automatically vi [CI system](../CI.rst). However, there are some +cases where we need to update them manually. This document describes how to do it. + +# Automated image cache and constraints refreshing in CI + +Our [CI system](../CI.rst) is build in the way that it self-maintains. Regular scheduled builds and +merges to `main` branch have separate maintenance step that take care about refreshing the cache that is +used to speed up our builds and to speed up rebuilding of [Breeze](../BREEZE.rst) images for development +purpose. This is all happening automatically, usually: + +* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch + after all tests succeeded in `main` merge or in `scheduled` build Review Comment: I realised this chapter was a little outdated an inacurate. Updated a lot of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
