Well that's the whole point about repeating "SemVer" enough times should be enough to distinct the feature (2.9.0) release from patchlevel (BTW. patchlevel name comes from SemVer so I am not sure if that changes much to mention it). But maybe another suggestion - I think maybe you can propose a PR how to distinguish those PRs in a clearer way Bolke to distinguish those ?
The release process - including the template of the vote email is in https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#prepare-vote-email-on-the-apache-airflow-release-candidat - and again Release Manager's work is really "mechanical" (including sending the email prepared from the procedure we have). So any of us can propose (and many times we did) a PR change to the process and templates sent - so if we think we can improve it and make it clearer - PR proposal to the process might be a good idea. J. On Thu, Jan 18, 2024 at 11:56 AM Bolke de Bruin <bdbr...@gmail.com> wrote: > Thanks Jarek, as mentioned in the other thread it seemed to break a > different pattern for me. I was aware of "what goes into the next release", > but the title seemed more indicative of roadmap items and thus geared to > consumers of a release. > > Earlier, it seems, I was helped by others when PRs were tagged for a > particular release so I was under the impression this felt under the > release manager's way of working. > > I suggested adding a label to the vote, like patch-release, and including > the link to the doc. That makes it easier for more occasional viewers. > > B. > > On Wed, 17 Jan 2024 at 13:42, Jarek Potiuk <ja...@potiuk.com> wrote: > >> I separate it out, because it seems that despite the efforts to explain >> and document how our releases work It's not clear even for the PMC chair, >> so likely it warrants a separate thread - also it will be easier to find it >> in the archives this way. >> >> I think this is an important topic that all maintainers should be aware >> of, so let me take a task to explain it in a longer email (and separate >> thread). >> >> I created it in a form of FAQ, because it seems that similar questions >> might be asked by others. >> >> *Do we have a process defined here?* >> >> Answering Bolke's question - who was apparently confused what our process >> is: >> >> > I see that some improvements to Airflow.io (two weeks ago) were not >> included and some provider updates neither. Haven't checked anything else >> yet. >> >> Apparently there is some confusion about the process, but yes we have a >> well defined and well tested (pretty much 4 years now) process that we >> follow., We follow it since I remember actually - it's been also done the >> same in 1.10 - but with some variations, Likely we do it the same way since >> the beginning of 2.0, but it has been refined and improved over time - by >> those who volunteered their time in the release process (a lot of ad-hoc >> discussion have been traditionally happening in #release-management slack >> channel) and as of few months ago we even documented it (It was in November >> 2023) - with this PR https://github.com/apache/airflow/pull/35245 >> >> It is currently described in a prominent place in our most important (and >> over the last year or so the README, we made the README pretty short and >> contains only super-important information on how Airflow is developed) >> README.md under *"What goes into the next release?"* chapter: >> >> >> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release >> . >> >> *How does the selection process for cherry-picking work?* >> >> In short (and this is the most important thing that every maintainer >> should be aware of): *those maintainers who think that issue should be >> included should mark it with the next (in this case 2.8.1) milestone*. >> It's up to individual maintainers who want to include certain changes to >> take care about it and mark the issues they think are bug fixes, to go into >> the next release >> >> This is the only thing that the maintainer has to do to get the PR >> proposed to be considered in the next patchlevel release. Sometimes - if >> controversial - maintainers discuss the proposals in #release-management >> channel, sometimes in #development or in the PR itself (especially if the >> release manager decides to not include it and changes the milestone (and >> explains why). >> >> *What's the release manager's role ?* >> >> Release manager's job is purely mechanical (as also mandated by the >> Apache Software Foundation release manager role description) to assess >> cherry-pick ability of those changes. Release manager - at the sole >> discretion and individual decision (this is the only place in the whole ASF >> setup where a single person has such power to make individual decisions) >> can reject some of those who other maintainers think should be included. >> But the release manager on his own does not make proposals on what should >> be included. >> >> *Is this process following the ASF rules?* >> >> I believe yes, The release manager's role is nicely described here: >> https://infra.apache.org/release-publishing.html#releasemanager. And >> there is a far more complete description here that describes the whole >> process https://www.apache.org/legal/release-policy.html#management - >> also mentioning that it's the PMC's responsibility (and particularly PMC >> chair's) to adhere to the process. >> >> *What's the role of individual maintainers?* >> >> The role of maintainers (collectively) to propose things for the next >> release. In our case it happens with setting the milestone on a PR. >> >> *When proposed PRs are rejected?* >> >> There are various reasons to reject those - if too complex to cherry-pick >> or when the release manager assesses it's a new feature, not a bugfix. >> Essentially (according to semver) when it comes to the user-facing changes, >> the PATCHLEVEL release should contain only bugfixes. and may contain docs >> changes if they are fixing/improving docs (not about new features) and also >> environment/build script changes (so non-user-facing changes) as they are >> pretty much always needed to keep the things nicely building - those are >> usually skipped from the changelog as non-user facing). >> >> *Why are provider changes not cherry-picked?* >> >> In our case - basically none of the provider changes are cherry-picked - >> unless they are needed to make the builds work well (sometimes happen). >> Providers are ALWAYS released from the latest main code, not from the >> v2-8-stable branch. In fact all the tests and ci checks for providers are >> skipped in the non-main (v2* branches). So yes - not seeing provider >> changes cherry-picked is absolutely expected. >> >> *Do we skip over some changes when releasing a patchlevel release? What's >> the purpose of patch-level releases?* >> >> Answering Bolke's question: >> >> > Is that intentional? Ie. is that the purpose of this release. Other >> big(ger) and more recent changes have been included hence asking. >> >> The purpose of that release is as described in SemVer - to give the users >> bugfix-only release that has no new features. Of course it's sometimes >> debatable whether changes are features/bugfixes, but we usually use >> #release-management to quickly chat about it, and eventually the release >> manager always makes a comment in the PR when the milestone is changed and >> explains the reasoning. >> >> Skipping is not intentional because we never "skip" things when >> cherry-picking, It's *reverse* - those maintainer who think that certain >> bug fixes (or internal changes or sometimes even feature changes that we >> classify really as "bugfix" SHOULD intentionally mark those PRs they want >> with 2.8.1 (or basically next patch-level) to be *included. * So there >> is no skipping, if maintainer did not deliberately mark PR as upcoming >> milestone, it will just not be included >> >> *Where do some maintainers know about it **from ? * >> >> Because the maintainers who actively participate - either acting as >> release managers (those who raised their hand and acted as release managers >> generally speaking) and those who wanted their changes to be part of the >> past releases have been doing it for years. For years this has been simply >> followed and discussed in #release-management channel and #development (and >> in devlist whenever there were new releases) and generally the maintainers >> who took part in those discussions and release process are aware of that - >> also many maintainers know the process as it "soaked" in when they were >> just watching. >> >> However recently we've made an attempt to document it (the PR above). So >> you could learn it by reading it (even if it does not have the whole >> context above). >> >> *Why do some people not know about it?* >> >> Not sure. Maybe because they were not interested and never asked? Maybe >> because there was never a long email about it at our devlist, or the >> documentation about it in README was too succinct? >> >> Or maybe we need a better way of communicating it - I am not sure. But I >> hope this email will clarify a lot of it, and will prove valuable when >> searching in devlist. >> >> Maybe even someone who manages to read it all will update the README >> description of ours to explain it better :) and maybe create a nice FAQ >> that we can put in our docs? >> >> I really hope this mail - even if long - will make people a bit more >> aware of the process, and if someone has an idea how the "collective" >> awareness can be improved - I think it's a good idea to send PR or email >> (or maybe even record a video :) ??) that will be a better way of >> communicating it. >> >> J. >> >> >> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bdbr...@gmail.com> wrote: >> >>> Just checking: >>> >>> I see that some improvements to Airflow.io (two weeks ago) were not >>> included and some provider updates neither. Haven't checked anything else >>> yet. >>> >>> Is that intentional? Ie. is that the purpose of this release. Other >>> big(ger) and more recent changes have been included hence asking. >>> >>> B. >>> >>> Sent from my iPhone >>> >>> > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote: >>> > >>> > +1 (binding): Checked all my changes, I ran airflow in a few >>> combinations >>> > (MySQL / Postgres Local/Celery executor. It looks and works well - run >>> a >>> > few dags and navigated through a number of screens. Checked licences, >>> > signatures, checksums, performed a "reproducible build" check and it >>> worked >>> > (with a small glitch explained below). >>> > >>> > BTW. The new "hatchling-built" package looks good and it nicely >>> installs >>> > airflow + extras (and it has a far better and cleaner set of extras - >>> > finally you can reproducibly install airflow with `all` extra if you >>> > ***REALLY*** want :). >>> > >>> > Reproducibility (also for other PMC members doing the check): I found a >>> > small glitch in the "reproducible" part of verifying the packages. >>> While >>> > wheel and sdist packages are exactly the same binary-wise, the >>> > source-tarball was binarry different for me. I decompressed it and >>> compared >>> > the content - they are identical - but there is one difference which I >>> > overlooked - the group permissions for files in Ephraim's tarball are >>> > different from mine. I have totally forgotten about the fact that umask >>> > might set different group/other permissions when checking out the files >>> > from git. Fix will be coming shortly - in the meantime I recommend >>> anyone >>> > who will be doing the comparison to uncompress both tarballs and >>> compare >>> > the contents with `diff -r`. >>> > >>> > >>> > >>> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi < >>> >> ephraimanier...@apache.org> wrote: >>> >> >>> >> Hey fellow Airflowers, >>> >> >>> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the >>> release, >>> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at >>> 10:30 >>> >> am UTC >>> >> until Friday, January 19, 2024, at 10:30 am UTC >>> >> < >>> >> >>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440 >>> >>> , >>> >> and until 3 binding +1 votes have been received. >>> >> >>> >> Status of testing of the release is kept at >>> >> https://github.com/apache/airflow/issues/36808 >>> >> >>> >> Consider this my (binding) +1. >>> >> >>> >> Airflow 2.8.1rc1 is available at: >>> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/ >>> >> >>> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes >>> with >>> >> INSTALL instructions. >>> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release. >>> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel >>> "binary" >>> >> release. >>> >> >>> >> Public keys are available at: >>> >> https://dist.apache.org/repos/dist/release/airflow/KEYS >>> >> >>> >> Please vote accordingly: >>> >> >>> >> [ ] +1 approve >>> >> [ ] +0 no opinion >>> >> [ ] -1 disapprove with the reason >>> >> >>> >> Only votes from PMC members are binding, but all members of the >>> community >>> >> are encouraged to test the release and vote with "(non-binding)". >>> >> >>> >> The test procedure for PMC members is described in: >>> >> >>> >> >>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members >>> >> >>> >> The test procedure for and Contributors who would like to test this >>> RC is >>> >> described in: >>> >> >>> >> >>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors >>> >> >>> >> >>> >> Please note that the version number excludes the `rcX` string, so >>> it's now >>> >> simply 2.8.1. This will allow us to rename the artifact without >>> modifying >>> >> the artifact checksums when we actually release. >>> >> >>> >> Release Notes: >>> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst >>> >> >>> >> Changes since 2.8.0: >>> >> >>> >> *Significant Changes* >>> >> >>> >> *Target version for core dependency ``pendulum`` package set to 3 >>> >> (#36281).* >>> >> Support for pendulum 2.1.2 will be saved for a while, presumably >>> until the >>> >> next feature version of Airflow. >>> >> It is advised to upgrade user code to use pendulum 3 as soon as >>> possible. >>> >> >>> >> *Airflow packaging specification follows modern Python packaging >>> standards >>> >> (#36537).* >>> >> We standardized Airflow dependency configuration to follow latest >>> >> development in Python packaging by >>> >> using ``pyproject.toml``. Airflow is now compliant with those accepted >>> >> PEPs: >>> >> >>> >> * `PEP-440 Version Identification and Dependency Specification < >>> >> https://www.python.org/dev/peps/pep-0440/>`__ >>> >> * `PEP-517 A build-system independent format for source trees < >>> >> https://www.python.org/dev/peps/pep-0517/>`__ >>> >> * `PEP-518 Specifying Minimum Build System Requirements for Python >>> Projects >>> >> <https://www.python.org/dev/peps/pep-0518/>`__ >>> >> * `PEP-561 Distributing and Packaging Type Information < >>> >> https://www.python.org/dev/peps/pep-0561/>`__ >>> >> * `PEP-621 Storing project metadata in pyproject.toml < >>> >> https://www.python.org/dev/peps/pep-0621/>`__ >>> >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel >>> based) >>> >> < >>> >> https://www.python.org/dev/peps/pep-0660/>`__ >>> >> * `PEP-685 Comparison of extra names for optional distribution >>> dependencies >>> >> <https://www.python.org/dev/peps/pep-0685/>`__ >>> >> >>> >> Also we implement multiple license files support coming from Draft, >>> not yet >>> >> accepted (but supported by hatchling) PEP: >>> >> * `PEP 639 Improving License Clarity with Better Package Metadata < >>> >> https://peps.python.org/pep-0639/>`__ >>> >> >>> >> This has almost no noticeable impact on users if they are using modern >>> >> Python packaging and development tools, generally >>> >> speaking Airflow should behave as it did before when installing it >>> from >>> >> PyPI and it should be much easier to install >>> >> it for development purposes using ``pip install -e ".[devel]"``. >>> >> >>> >> The differences from the user side are: >>> >> >>> >> * Airflow extras now get extras normalized to ``-`` (following >>> PEP-685) >>> >> instead of ``_`` and ``.`` >>> >> (as it was before in some extras). When you install airflow with such >>> >> extras (for example ``dbt.core`` or >>> >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``. >>> >> >>> >> In most modern tools this will work in backwards-compatible way, but >>> in >>> >> some old version of those tools you might need to >>> >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the >>> >> extra you are installing does not exist - but usually >>> >> this warning is harmless and the extra is installed anyway. It is, >>> however, >>> >> recommended to change to use ``-`` in extras in your dependency >>> >> specifications for all Airflow extras. >>> >> >>> >> * Released airflow package does not contain ``devel``, ``devel-*``, >>> ``doc`` >>> >> and ``doc-gen`` extras. >>> >> Those extras are only available when you install Airflow from >>> sources in >>> >> ``--editable`` mode. This is >>> >> because those extras are only used for development and documentation >>> >> building purposes and are not needed >>> >> when you install Airflow for production use. Those dependencies had >>> >> unspecified and varying behaviour for >>> >> released packages anyway and you were not supposed to use them in >>> >> released packages. >>> >> >>> >> * The ``all`` and ``all-*`` extras were not always working correctly >>> when >>> >> installing Airflow using constraints >>> >> because they were also considered as development-only dependencies. >>> With >>> >> this change, those dependencies are >>> >> now properly handling constraints and they will install properly with >>> >> constraints, pulling the right set >>> >> of providers and dependencies when constraints are used. >>> >> >>> >> *Graphviz dependency is now an optional one, not required one >>> (#36647).* >>> >> The ``graphviz`` dependency has been problematic as Airflow required >>> >> dependency - especially for >>> >> ARM-based installations. Graphviz packages require binary graphviz >>> >> libraries - which is already a >>> >> limitation, but they also require to install graphviz Python bindings >>> to be >>> >> build and installed. >>> >> This does not work for older Linux installation but - more >>> importantly - >>> >> when you try to install >>> >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the >>> packages >>> >> fail to install because >>> >> Python bindings compilation for M1 can only work for Python 3.10+. >>> >> >>> >> This is not a breaking change technically - the CLIs to render the >>> DAGs is >>> >> still there and IF you >>> >> already have graphviz installed, it will continue working as it did >>> before. >>> >> The only problem when it >>> >> does not work is where you do not have graphviz installed it will >>> raise an >>> >> error and inform that you need it. >>> >> >>> >> Graphviz will remain to be installed for most users: >>> >> >>> >> * the Airflow Image will still contain graphviz library, because >>> >> it is added there as extra >>> >> * when previous version of Airflow has been installed already, then >>> >> graphviz library is already installed there and Airflow will >>> >> continue working as it did >>> >> >>> >> The only change will be a new installation of new version of Airflow >>> from >>> >> the scratch, where graphviz will >>> >> need to be specified as extra or installed separately in order to >>> enable >>> >> DAG rendering option. >>> >> >>> >> *Bug Fixes* >>> >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800) >>> >> - Fix Callback exception when a removed task is the last one in the >>> >> ``taskinstance`` list (#36693) >>> >> - Allow anonymous user edit/show resource when set >>> >> ``AUTH_ROLE_PUBLIC=admin`` (#36750) >>> >> - Better error message when sqlite URL uses relative path (#36774) >>> >> - Explicit string cast required to force integer-type run_ids to be >>> passed >>> >> as strings instead of integers (#36756) >>> >> - Add log lookup exception for empty ``op`` subtypes (#35536) >>> >> - Remove unused index on task instance (#36737) >>> >> - Fix check on subclass for ``typing.Union`` in >>> ``_infer_multiple_outputs`` >>> >> for Python 3.10+ (#36728) >>> >> - Make sure ``multiple_outputs`` is inferred correctly even when using >>> >> ``TypedDict`` (#36652) >>> >> - Add back FAB constant in legacy security manager (#36719) >>> >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712) >>> >> - Do not let ``EventsTimetable`` schedule past events if >>> ``catchup=False`` >>> >> (#36134) >>> >> - Support encryption for triggers parameters (#36492) >>> >> - Fix the type hint for ``tis_query`` in ``_process_executor_events`` >>> >> (#36655) >>> >> - Redirect to index when user does not have permission to access a >>> page >>> >> (#36623) >>> >> - Avoid using dict as default value in ``call_regular_interval`` >>> (#36608) >>> >> - Remove option to set a task instance to running state in UI (#36518) >>> >> - Fix details tab not showing when using dynamic task mapping (#36522) >>> >> - Raise error when ``DagRun`` fails while running ``dag test`` >>> (#36517) >>> >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch >>> (#36502) >>> >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401) >>> >> - Fix get_leaves calculation for teardown in nested group (#36456) >>> >> - Stop serializing timezone-naive datetime to timezone-aware datetime >>> with >>> >> UTC tz (#36379) >>> >> - Make ``kubernetes`` decorator type annotation consistent with >>> operator >>> >> (#36405) >>> >> - Fix Webserver returning 500 for POST requests to >>> ``api/dag/*/dagrun`` >>> >> from anonymous user (#36275) >>> >> - Fix the required access for get_variable endpoint (#36396) >>> >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370) >>> >> - Fix AirflowSkipException message raised by BashOperator (#36354) >>> >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361) >>> >> - Increase width of execution_date input in trigger.html (#36278) >>> >> - Fix logging for pausing DAG (#36182) >>> >> - Stop deserializing pickle when enable_xcom_pickling is False >>> (#36255) >>> >> - Check DAG read permission before accessing DAG code (#36257) >>> >> - Enable mark task as failed/success always (#36254) >>> >> - Create latest log dir symlink as relative link (#36019) >>> >> - Fix Python-based decorators templating (#36103) >>> >> >>> >> *Miscellaneous* >>> >> - Rename concurrency label to max active tasks (#36691) >>> >> - Restore function scoped ``httpx`` import in file_task_handler for >>> >> performance (#36753) >>> >> - Add support of Pendulum 3 (#36281) >>> >> - Standardize airflow build process and switch to Hatchling build >>> backend >>> >> (#36537) >>> >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697) >>> >> - Make ``graphviz`` dependency optional (#36647) >>> >> - Announce MSSQL support end in Airflow 2.9.0, add migration script >>> hints >>> >> (#36509) >>> >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow >>> >> (#36698) >>> >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www`` >>> (#36700) >>> >> - Provide the logger_name param to base hook in order to override the >>> >> logger name (#36674) >>> >> - Fix run type icon alignment with run type text (#36616) >>> >> - Follow BaseHook connection fields method signature in FSHook >>> (#36444) >>> >> - Remove redundant ``docker`` decorator type annotations (#36406) >>> >> - Straighten typing in workday timetable (#36296) >>> >> - Use ``batch_is_authorized_dag`` to check if user has permission to >>> read >>> >> DAGs (#36279) >>> >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags >>> in >>> >> get_dag_warnings (#36256) >>> >> >>> >> *Doc Only Changes* >>> >> - Metrics tagging documentation (#36627) >>> >> - In docs use logical_date instead of deprecated execution_date >>> (#36654) >>> >> - Add section about live-upgrading Airflow (#36637) >>> >> - Replace ``numpy`` example with practical exercise demonstrating >>> top-level >>> >> code (#35097) >>> >> - Improve and add more complete description in the architecture >>> diagrams >>> >> (#36513) >>> >> - Improve the error message displayed when there is a webserver error >>> >> (#36570) >>> >> - Update ``dags.rst`` with information on DAG pausing (#36540) >>> >> - Update installation prerequisites after upgrading to Debian Bookworm >>> >> (#36521) >>> >> - Add description on the ways how users should approach DB monitoring >>> >> (#36483) >>> >> - Add branching based on mapped task group example to >>> >> dynamic-task-mapping.rst (#36480) >>> >> - Add further details to replacement documentation (#36485) >>> >> - Use cards when describing priority weighting methods (#36411) >>> >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404) >>> >> - Update admonitions in Python operator doc to reflect sentiment >>> (#36340) >>> >> - Improve audit_logs.rst (#36213) >>> >> - Remove Redshift mention from the list of managed Postgres backends >>> >> (#36217) >>> >> >>> >> Cheers, >>> >> Ephraim >>> >> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> For additional commands, e-mail: dev-h...@airflow.apache.org >>> >>> > > -- > > -- > Bolke de Bruin > bdbr...@gmail.com >
