Thanks Jarek, as mentioned in the other thread it seemed to break a different pattern for me. I was aware of "what goes into the next release", but the title seemed more indicative of roadmap items and thus geared to consumers of a release.
Earlier, it seems, I was helped by others when PRs were tagged for a particular release so I was under the impression this felt under the release manager's way of working. I suggested adding a label to the vote, like patch-release, and including the link to the doc. That makes it easier for more occasional viewers. B. On Wed, 17 Jan 2024 at 13:42, Jarek Potiuk <ja...@potiuk.com> wrote: > I separate it out, because it seems that despite the efforts to explain > and document how our releases work It's not clear even for the PMC chair, > so likely it warrants a separate thread - also it will be easier to find it > in the archives this way. > > I think this is an important topic that all maintainers should be aware > of, so let me take a task to explain it in a longer email (and separate > thread). > > I created it in a form of FAQ, because it seems that similar questions > might be asked by others. > > *Do we have a process defined here?* > > Answering Bolke's question - who was apparently confused what our process > is: > > > I see that some improvements to Airflow.io (two weeks ago) were not > included and some provider updates neither. Haven't checked anything else > yet. > > Apparently there is some confusion about the process, but yes we have a > well defined and well tested (pretty much 4 years now) process that we > follow., We follow it since I remember actually - it's been also done the > same in 1.10 - but with some variations, Likely we do it the same way since > the beginning of 2.0, but it has been refined and improved over time - by > those who volunteered their time in the release process (a lot of ad-hoc > discussion have been traditionally happening in #release-management slack > channel) and as of few months ago we even documented it (It was in November > 2023) - with this PR https://github.com/apache/airflow/pull/35245 > > It is currently described in a prominent place in our most important (and > over the last year or so the README, we made the README pretty short and > contains only super-important information on how Airflow is developed) > README.md under *"What goes into the next release?"* chapter: > > > https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release > . > > *How does the selection process for cherry-picking work?* > > In short (and this is the most important thing that every maintainer > should be aware of): *those maintainers who think that issue should be > included should mark it with the next (in this case 2.8.1) milestone*. > It's up to individual maintainers who want to include certain changes to > take care about it and mark the issues they think are bug fixes, to go into > the next release > > This is the only thing that the maintainer has to do to get the PR > proposed to be considered in the next patchlevel release. Sometimes - if > controversial - maintainers discuss the proposals in #release-management > channel, sometimes in #development or in the PR itself (especially if the > release manager decides to not include it and changes the milestone (and > explains why). > > *What's the release manager's role ?* > > Release manager's job is purely mechanical (as also mandated by the Apache > Software Foundation release manager role description) to assess cherry-pick > ability of those changes. Release manager - at the sole discretion and > individual decision (this is the only place in the whole ASF setup where a > single person has such power to make individual decisions) can reject some > of those who other maintainers think should be included. But the release > manager on his own does not make proposals on what should be included. > > *Is this process following the ASF rules?* > > I believe yes, The release manager's role is nicely described here: > https://infra.apache.org/release-publishing.html#releasemanager. And > there is a far more complete description here that describes the whole > process https://www.apache.org/legal/release-policy.html#management - > also mentioning that it's the PMC's responsibility (and particularly PMC > chair's) to adhere to the process. > > *What's the role of individual maintainers?* > > The role of maintainers (collectively) to propose things for the next > release. In our case it happens with setting the milestone on a PR. > > *When proposed PRs are rejected?* > > There are various reasons to reject those - if too complex to cherry-pick > or when the release manager assesses it's a new feature, not a bugfix. > Essentially (according to semver) when it comes to the user-facing changes, > the PATCHLEVEL release should contain only bugfixes. and may contain docs > changes if they are fixing/improving docs (not about new features) and also > environment/build script changes (so non-user-facing changes) as they are > pretty much always needed to keep the things nicely building - those are > usually skipped from the changelog as non-user facing). > > *Why are provider changes not cherry-picked?* > > In our case - basically none of the provider changes are cherry-picked - > unless they are needed to make the builds work well (sometimes happen). > Providers are ALWAYS released from the latest main code, not from the > v2-8-stable branch. In fact all the tests and ci checks for providers are > skipped in the non-main (v2* branches). So yes - not seeing provider > changes cherry-picked is absolutely expected. > > *Do we skip over some changes when releasing a patchlevel release? What's > the purpose of patch-level releases?* > > Answering Bolke's question: > > > Is that intentional? Ie. is that the purpose of this release. Other > big(ger) and more recent changes have been included hence asking. > > The purpose of that release is as described in SemVer - to give the users > bugfix-only release that has no new features. Of course it's sometimes > debatable whether changes are features/bugfixes, but we usually use > #release-management to quickly chat about it, and eventually the release > manager always makes a comment in the PR when the milestone is changed and > explains the reasoning. > > Skipping is not intentional because we never "skip" things when > cherry-picking, It's *reverse* - those maintainer who think that certain > bug fixes (or internal changes or sometimes even feature changes that we > classify really as "bugfix" SHOULD intentionally mark those PRs they want > with 2.8.1 (or basically next patch-level) to be *included. * So there is > no skipping, if maintainer did not deliberately mark PR as upcoming > milestone, it will just not be included > > *Where do some maintainers know about it **from ? * > > Because the maintainers who actively participate - either acting as > release managers (those who raised their hand and acted as release managers > generally speaking) and those who wanted their changes to be part of the > past releases have been doing it for years. For years this has been simply > followed and discussed in #release-management channel and #development (and > in devlist whenever there were new releases) and generally the maintainers > who took part in those discussions and release process are aware of that - > also many maintainers know the process as it "soaked" in when they were > just watching. > > However recently we've made an attempt to document it (the PR above). So > you could learn it by reading it (even if it does not have the whole > context above). > > *Why do some people not know about it?* > > Not sure. Maybe because they were not interested and never asked? Maybe > because there was never a long email about it at our devlist, or the > documentation about it in README was too succinct? > > Or maybe we need a better way of communicating it - I am not sure. But I > hope this email will clarify a lot of it, and will prove valuable when > searching in devlist. > > Maybe even someone who manages to read it all will update the README > description of ours to explain it better :) and maybe create a nice FAQ > that we can put in our docs? > > I really hope this mail - even if long - will make people a bit more aware > of the process, and if someone has an idea how the "collective" awareness > can be improved - I think it's a good idea to send PR or email (or maybe > even record a video :) ??) that will be a better way of communicating it. > > J. > > > On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Just checking: >> >> I see that some improvements to Airflow.io (two weeks ago) were not >> included and some provider updates neither. Haven't checked anything else >> yet. >> >> Is that intentional? Ie. is that the purpose of this release. Other >> big(ger) and more recent changes have been included hence asking. >> >> B. >> >> Sent from my iPhone >> >> > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > +1 (binding): Checked all my changes, I ran airflow in a few >> combinations >> > (MySQL / Postgres Local/Celery executor. It looks and works well - run a >> > few dags and navigated through a number of screens. Checked licences, >> > signatures, checksums, performed a "reproducible build" check and it >> worked >> > (with a small glitch explained below). >> > >> > BTW. The new "hatchling-built" package looks good and it nicely installs >> > airflow + extras (and it has a far better and cleaner set of extras - >> > finally you can reproducibly install airflow with `all` extra if you >> > ***REALLY*** want :). >> > >> > Reproducibility (also for other PMC members doing the check): I found a >> > small glitch in the "reproducible" part of verifying the packages. While >> > wheel and sdist packages are exactly the same binary-wise, the >> > source-tarball was binarry different for me. I decompressed it and >> compared >> > the content - they are identical - but there is one difference which I >> > overlooked - the group permissions for files in Ephraim's tarball are >> > different from mine. I have totally forgotten about the fact that umask >> > might set different group/other permissions when checking out the files >> > from git. Fix will be coming shortly - in the meantime I recommend >> anyone >> > who will be doing the comparison to uncompress both tarballs and compare >> > the contents with `diff -r`. >> > >> > >> > >> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi < >> >> ephraimanier...@apache.org> wrote: >> >> >> >> Hey fellow Airflowers, >> >> >> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the >> release, >> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at >> 10:30 >> >> am UTC >> >> until Friday, January 19, 2024, at 10:30 am UTC >> >> < >> >> >> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440 >> >>> , >> >> and until 3 binding +1 votes have been received. >> >> >> >> Status of testing of the release is kept at >> >> https://github.com/apache/airflow/issues/36808 >> >> >> >> Consider this my (binding) +1. >> >> >> >> Airflow 2.8.1rc1 is available at: >> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/ >> >> >> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes >> with >> >> INSTALL instructions. >> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release. >> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel >> "binary" >> >> release. >> >> >> >> Public keys are available at: >> >> https://dist.apache.org/repos/dist/release/airflow/KEYS >> >> >> >> Please vote accordingly: >> >> >> >> [ ] +1 approve >> >> [ ] +0 no opinion >> >> [ ] -1 disapprove with the reason >> >> >> >> Only votes from PMC members are binding, but all members of the >> community >> >> are encouraged to test the release and vote with "(non-binding)". >> >> >> >> The test procedure for PMC members is described in: >> >> >> >> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members >> >> >> >> The test procedure for and Contributors who would like to test this RC >> is >> >> described in: >> >> >> >> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors >> >> >> >> >> >> Please note that the version number excludes the `rcX` string, so it's >> now >> >> simply 2.8.1. This will allow us to rename the artifact without >> modifying >> >> the artifact checksums when we actually release. >> >> >> >> Release Notes: >> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst >> >> >> >> Changes since 2.8.0: >> >> >> >> *Significant Changes* >> >> >> >> *Target version for core dependency ``pendulum`` package set to 3 >> >> (#36281).* >> >> Support for pendulum 2.1.2 will be saved for a while, presumably until >> the >> >> next feature version of Airflow. >> >> It is advised to upgrade user code to use pendulum 3 as soon as >> possible. >> >> >> >> *Airflow packaging specification follows modern Python packaging >> standards >> >> (#36537).* >> >> We standardized Airflow dependency configuration to follow latest >> >> development in Python packaging by >> >> using ``pyproject.toml``. Airflow is now compliant with those accepted >> >> PEPs: >> >> >> >> * `PEP-440 Version Identification and Dependency Specification < >> >> https://www.python.org/dev/peps/pep-0440/>`__ >> >> * `PEP-517 A build-system independent format for source trees < >> >> https://www.python.org/dev/peps/pep-0517/>`__ >> >> * `PEP-518 Specifying Minimum Build System Requirements for Python >> Projects >> >> <https://www.python.org/dev/peps/pep-0518/>`__ >> >> * `PEP-561 Distributing and Packaging Type Information < >> >> https://www.python.org/dev/peps/pep-0561/>`__ >> >> * `PEP-621 Storing project metadata in pyproject.toml < >> >> https://www.python.org/dev/peps/pep-0621/>`__ >> >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel >> based) >> >> < >> >> https://www.python.org/dev/peps/pep-0660/>`__ >> >> * `PEP-685 Comparison of extra names for optional distribution >> dependencies >> >> <https://www.python.org/dev/peps/pep-0685/>`__ >> >> >> >> Also we implement multiple license files support coming from Draft, >> not yet >> >> accepted (but supported by hatchling) PEP: >> >> * `PEP 639 Improving License Clarity with Better Package Metadata < >> >> https://peps.python.org/pep-0639/>`__ >> >> >> >> This has almost no noticeable impact on users if they are using modern >> >> Python packaging and development tools, generally >> >> speaking Airflow should behave as it did before when installing it from >> >> PyPI and it should be much easier to install >> >> it for development purposes using ``pip install -e ".[devel]"``. >> >> >> >> The differences from the user side are: >> >> >> >> * Airflow extras now get extras normalized to ``-`` (following PEP-685) >> >> instead of ``_`` and ``.`` >> >> (as it was before in some extras). When you install airflow with such >> >> extras (for example ``dbt.core`` or >> >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``. >> >> >> >> In most modern tools this will work in backwards-compatible way, but in >> >> some old version of those tools you might need to >> >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the >> >> extra you are installing does not exist - but usually >> >> this warning is harmless and the extra is installed anyway. It is, >> however, >> >> recommended to change to use ``-`` in extras in your dependency >> >> specifications for all Airflow extras. >> >> >> >> * Released airflow package does not contain ``devel``, ``devel-*``, >> ``doc`` >> >> and ``doc-gen`` extras. >> >> Those extras are only available when you install Airflow from sources >> in >> >> ``--editable`` mode. This is >> >> because those extras are only used for development and documentation >> >> building purposes and are not needed >> >> when you install Airflow for production use. Those dependencies had >> >> unspecified and varying behaviour for >> >> released packages anyway and you were not supposed to use them in >> >> released packages. >> >> >> >> * The ``all`` and ``all-*`` extras were not always working correctly >> when >> >> installing Airflow using constraints >> >> because they were also considered as development-only dependencies. >> With >> >> this change, those dependencies are >> >> now properly handling constraints and they will install properly with >> >> constraints, pulling the right set >> >> of providers and dependencies when constraints are used. >> >> >> >> *Graphviz dependency is now an optional one, not required one >> (#36647).* >> >> The ``graphviz`` dependency has been problematic as Airflow required >> >> dependency - especially for >> >> ARM-based installations. Graphviz packages require binary graphviz >> >> libraries - which is already a >> >> limitation, but they also require to install graphviz Python bindings >> to be >> >> build and installed. >> >> This does not work for older Linux installation but - more importantly >> - >> >> when you try to install >> >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the >> packages >> >> fail to install because >> >> Python bindings compilation for M1 can only work for Python 3.10+. >> >> >> >> This is not a breaking change technically - the CLIs to render the >> DAGs is >> >> still there and IF you >> >> already have graphviz installed, it will continue working as it did >> before. >> >> The only problem when it >> >> does not work is where you do not have graphviz installed it will >> raise an >> >> error and inform that you need it. >> >> >> >> Graphviz will remain to be installed for most users: >> >> >> >> * the Airflow Image will still contain graphviz library, because >> >> it is added there as extra >> >> * when previous version of Airflow has been installed already, then >> >> graphviz library is already installed there and Airflow will >> >> continue working as it did >> >> >> >> The only change will be a new installation of new version of Airflow >> from >> >> the scratch, where graphviz will >> >> need to be specified as extra or installed separately in order to >> enable >> >> DAG rendering option. >> >> >> >> *Bug Fixes* >> >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800) >> >> - Fix Callback exception when a removed task is the last one in the >> >> ``taskinstance`` list (#36693) >> >> - Allow anonymous user edit/show resource when set >> >> ``AUTH_ROLE_PUBLIC=admin`` (#36750) >> >> - Better error message when sqlite URL uses relative path (#36774) >> >> - Explicit string cast required to force integer-type run_ids to be >> passed >> >> as strings instead of integers (#36756) >> >> - Add log lookup exception for empty ``op`` subtypes (#35536) >> >> - Remove unused index on task instance (#36737) >> >> - Fix check on subclass for ``typing.Union`` in >> ``_infer_multiple_outputs`` >> >> for Python 3.10+ (#36728) >> >> - Make sure ``multiple_outputs`` is inferred correctly even when using >> >> ``TypedDict`` (#36652) >> >> - Add back FAB constant in legacy security manager (#36719) >> >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712) >> >> - Do not let ``EventsTimetable`` schedule past events if >> ``catchup=False`` >> >> (#36134) >> >> - Support encryption for triggers parameters (#36492) >> >> - Fix the type hint for ``tis_query`` in ``_process_executor_events`` >> >> (#36655) >> >> - Redirect to index when user does not have permission to access a page >> >> (#36623) >> >> - Avoid using dict as default value in ``call_regular_interval`` >> (#36608) >> >> - Remove option to set a task instance to running state in UI (#36518) >> >> - Fix details tab not showing when using dynamic task mapping (#36522) >> >> - Raise error when ``DagRun`` fails while running ``dag test`` (#36517) >> >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch >> (#36502) >> >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401) >> >> - Fix get_leaves calculation for teardown in nested group (#36456) >> >> - Stop serializing timezone-naive datetime to timezone-aware datetime >> with >> >> UTC tz (#36379) >> >> - Make ``kubernetes`` decorator type annotation consistent with >> operator >> >> (#36405) >> >> - Fix Webserver returning 500 for POST requests to ``api/dag/*/dagrun`` >> >> from anonymous user (#36275) >> >> - Fix the required access for get_variable endpoint (#36396) >> >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370) >> >> - Fix AirflowSkipException message raised by BashOperator (#36354) >> >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361) >> >> - Increase width of execution_date input in trigger.html (#36278) >> >> - Fix logging for pausing DAG (#36182) >> >> - Stop deserializing pickle when enable_xcom_pickling is False (#36255) >> >> - Check DAG read permission before accessing DAG code (#36257) >> >> - Enable mark task as failed/success always (#36254) >> >> - Create latest log dir symlink as relative link (#36019) >> >> - Fix Python-based decorators templating (#36103) >> >> >> >> *Miscellaneous* >> >> - Rename concurrency label to max active tasks (#36691) >> >> - Restore function scoped ``httpx`` import in file_task_handler for >> >> performance (#36753) >> >> - Add support of Pendulum 3 (#36281) >> >> - Standardize airflow build process and switch to Hatchling build >> backend >> >> (#36537) >> >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697) >> >> - Make ``graphviz`` dependency optional (#36647) >> >> - Announce MSSQL support end in Airflow 2.9.0, add migration script >> hints >> >> (#36509) >> >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow >> >> (#36698) >> >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www`` >> (#36700) >> >> - Provide the logger_name param to base hook in order to override the >> >> logger name (#36674) >> >> - Fix run type icon alignment with run type text (#36616) >> >> - Follow BaseHook connection fields method signature in FSHook (#36444) >> >> - Remove redundant ``docker`` decorator type annotations (#36406) >> >> - Straighten typing in workday timetable (#36296) >> >> - Use ``batch_is_authorized_dag`` to check if user has permission to >> read >> >> DAGs (#36279) >> >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags >> in >> >> get_dag_warnings (#36256) >> >> >> >> *Doc Only Changes* >> >> - Metrics tagging documentation (#36627) >> >> - In docs use logical_date instead of deprecated execution_date >> (#36654) >> >> - Add section about live-upgrading Airflow (#36637) >> >> - Replace ``numpy`` example with practical exercise demonstrating >> top-level >> >> code (#35097) >> >> - Improve and add more complete description in the architecture >> diagrams >> >> (#36513) >> >> - Improve the error message displayed when there is a webserver error >> >> (#36570) >> >> - Update ``dags.rst`` with information on DAG pausing (#36540) >> >> - Update installation prerequisites after upgrading to Debian Bookworm >> >> (#36521) >> >> - Add description on the ways how users should approach DB monitoring >> >> (#36483) >> >> - Add branching based on mapped task group example to >> >> dynamic-task-mapping.rst (#36480) >> >> - Add further details to replacement documentation (#36485) >> >> - Use cards when describing priority weighting methods (#36411) >> >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404) >> >> - Update admonitions in Python operator doc to reflect sentiment >> (#36340) >> >> - Improve audit_logs.rst (#36213) >> >> - Remove Redshift mention from the list of managed Postgres backends >> >> (#36217) >> >> >> >> Cheers, >> >> Ephraim >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >> -- -- Bolke de Bruin bdbr...@gmail.com
