Slim images are now available too: - docker pull apache/airflow:slim-2.10.0rc1-python3.8 - docker pull apache/airflow:slim-2.10.0rc1-python3.9 - docker pull apache/airflow:slim-2.10.0rc1-python3.10 - docker pull apache/airflow:slim-2.10.0rc1-python3.11 - docker pull apache/airflow:slim-2.10.0rc1-python3.12
On Mon, 12 Aug 2024 at 23:24, Kaxil Naik <kaxiln...@gmail.com> wrote: > non-slim images has been pushed and are now available. Slim images are in > progress. > > > https://hub.docker.com/repository/docker/apache/airflow/tags?page=&page_size=&ordering=&name=2.10.0rc1 > > On Mon, 12 Aug 2024 at 20:07, Utkarsh Sharma > <utkarsh.sha...@astronomer.io.invalid> wrote: > >> Dear Airflow Community, >> >> We are currently experiencing some CI-related issues, which will >> temporarily delay the availability of the Docker image for Airflow 2.10.0 >> RC1. We anticipate that it will take a few hours to resolve these issues. >> >> Thanks, >> >> Utkarsh Sharma >> >> On Tue, Aug 13, 2024 at 12:33 AM Utkarsh Sharma < >> utkarsh.sha...@astronomer.io> wrote: >> >> > Hey fellow Airflowers, >> > >> > I have cut Airflow 2.10.0rc1. This email is calling a vote on the >> release, >> > which will last at least 72 hours, from Monday, August 12, 2024 at >> 6:00 pm >> > UTC >> > until Wednesday, August 15, 2024 at 6:00 pm UTC >> > < >> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240815T1800&p1=1440 >> >, >> > and until 3 binding +1 votes have been received. >> > >> > >> > >> https://www.timeanddate.com/worldclock/fixedtime.html\?msg\=8\&iso\=20211011T1600\&p1\=1440 >> > >> > Consider this my (non-binding) +1. As I’m not a member of the PMC, >> Ephraim >> > signed the distribution. >> > >> > Airflow 2.10.0rc1 is available at: >> > https://dist.apache.org/repos/dist/dev/airflow/2.10.0rc1/ >> > >> > *apache-airflow-2.10.0-source.tar.gz* is a source release that comes >> with >> > INSTALL instructions. >> > *apache-airflow-2.10.0.tar.gz* is the binary Python "sdist" release. >> > *apache_airflow-2.10.0-py3-none-any.whl* is the binary Python wheel >> > "binary" release. >> > >> > Public keys are available at: >> > https://dist.apache.org/repos/dist/release/airflow/KEYS >> > >> > Please vote accordingly: >> > >> > [ ] +1 approve >> > [ ] +0 no opinion >> > [ ] -1 disapprove with the reason >> > >> > Only votes from PMC members are binding, but all members of the >> community >> > are encouraged to test the release and vote with "(non-binding)". >> > >> > The test procedure for PMC members is described in: >> > >> > >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members >> > >> > The test procedure for contributors and members of the community who >> would >> > like to test this RC is described in: >> > >> > >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors >> > >> > >> > Please note that the version number excludes the `rcX` string, so it's >> now >> > simply 2.10.0. This will allow us to rename the artifact without >> modifying >> > the artifact checksums when we actually release. >> > >> > Release Notes: >> > https://github.com/apache/airflow/blob/2.10.0rc1/RELEASE_NOTES.rst >> > >> > For information on what goes into a release please see: >> > >> https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md >> > >> > *Changes since 2.9.3:* >> > >> > *Significant Changes* >> > >> > *Datasets no longer trigger inactive DAGs (#38891)* >> > >> > Previously, when a DAG is paused or removed, incoming dataset events >> would >> > still >> > trigger it, and the DAG would run when it is unpaused or added back in a >> > DAG >> > file. This has been changed; a DAG's dataset schedule can now only be >> > satisfied >> > by events that occur when the DAG is active. While this is a breaking >> > change, >> > the previous behavior is considered a bug. >> > >> > The behavior of time-based scheduling is unchanged, including the >> > timetable part >> > of ``DatasetOrTimeSchedule``. >> > >> > *``try_number`` is no longer incremented during task execution (#39336)* >> > >> > Previously, the try number (``try_number``) was incremented at the >> > beginning of task execution on the worker. This was problematic for many >> > reasons. >> > For one it meant that the try number was incremented when it was not >> > supposed to, namely when resuming from reschedule or deferral. And it >> also >> > resulted in >> > the try number being "wrong" when the task had not yet started. The >> > workarounds for these two issues caused a lot of confusion. >> > >> > Now, instead, the try number for a task run is determined at the time >> the >> > task is scheduled, and does not change in flight, and it is never >> > decremented. >> > So after the task runs, the observed try number remains the same as it >> was >> > when the task was running; only when there is a "new try" will the try >> > number be incremented again. >> > >> > One consequence of this change is, if users were "manually" running >> tasks >> > (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks >> > run``), >> > try number will no longer be incremented. Airflow assumes that tasks are >> > always run after being scheduled by the scheduler, so we do not regard >> this >> > as a breaking change. >> > >> > *``/logout`` endpoint in FAB Auth Manager is now CSRF protected >> (#40145)* >> > >> > The ``/logout`` endpoint's method in FAB Auth Manager has been changed >> > from ``GET`` to ``POST`` in all existing >> > AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, >> > ``AuthOIDView``, ``AuthRemoteUserView``), and >> > now includes CSRF protection to enhance security and prevent >> unauthorized >> > logouts. >> > >> > *OpenTelemetry Traces for Apache Airflow (#37948).* >> > >> > This new feature adds capability for Apache Airflow to emit 1) airflow >> > system traces of scheduler, >> > triggerer, executor, processor 2) DAG run traces for deployed DAG runs >> in >> > OpenTelemetry format. Previously, only metrics were supported which >> emitted >> > metrics in OpenTelemetry. >> > This new feature will add richer data for users to use OpenTelemetry >> > standard to emit and send their trace data to OTLP compatible endpoints. >> > >> > *Decorator for Task Flow ``(@skip_if, @run_if)`` to make it simple to >> > apply whether or not to skip a Task. (#41116)* >> > >> > This feature adds a decorator to make it simple to skip a Task. >> > >> > *Using Multiple Executors Concurrently (#40701)* >> > >> > Previously known as hybrid executors, this new feature allows Airflow to >> > use multiple executors concurrently. DAGs, or even individual tasks, >> can be >> > configured >> > to use a specific executor that suits its needs best. A single DAG can >> > contain tasks all using different executors. Please see the Airflow >> > documentation for >> > more details. Note: This feature is still experimental. See >> `documentation >> > on Executor < >> > >> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently >> >`_ >> > for a more detailed description. >> > >> > *Scarf based telemetry: Does Airflow collect any telemetry data? >> (#39510)* >> > >> > Airflow integrates Scarf to collect basic usage data during operation. >> > Deployments can opt-out of data collection by setting the >> > ``[usage_data_collection]enabled`` option to False, or the >> > SCARF_ANALYTICS=false environment variable. >> > See `FAQ on this < >> > >> https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data >> >`_ >> > for more information. >> > >> > >> > *New Features* >> > >> > - AIP-61 Hybrid Execution (`AIP-61 < >> > >> https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22 >> > >`_) >> > - AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 < >> > >> https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22 >> > >`_) >> > - AIP-64 TaskInstance Try History (`AIP-64 < >> > >> https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22 >> > >`_) >> > - AIP-44 Internal API (`AIP-44 < >> > >> https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed >> > >`_) >> > - Enable ending the task directly from the triggerer without going into >> > the worker. (#40084) >> > - Extend dataset dependencies (#40868) >> > - Feature/add token authentication to internal api (#40899) >> > - Add DatasetAlias to support dynamic Dataset Event Emission and Dataset >> > Creation (#40478) >> > - Add example DAGs for inlet_events (#39893) >> > - Implement ``accessors`` to read dataset events defined as inlet >> (#39367) >> > - Decorator for Task Flow, to make it simple to apply whether or not to >> > skip a Task. (#41116) >> > - Add start execution from triggerer support to dynamic task mapping >> > (#39912) >> > - Add try_number to log table (#40739) >> > - Added ds_format_locale method in macros which allows localizing >> datetime >> > formatting using Babel (#40746) >> > - Add DatasetAlias to support dynamic Dataset Event Emission and Dataset >> > Creation (#40478, #40723, #40809, #41264, #40830, #40693, #41302) >> > - Use sentinel to mark dag as removed on re-serialization (#39825) >> > - Add parameter for the last number of queries to the DB in DAG file >> > processing stats (#40323) >> > - Add prototype version dark mode for Airflow UI (#39355) >> > - Add ability to mark some tasks as successful in ``dag test`` (#40010) >> > - Allow use of callable for template_fields (#37028) >> > - Filter running/failed and active/paused dags on the home page(#39701) >> > - Add metrics about task CPU and memory usage (#39650) >> > - UI changes for DAG Re-parsing feature (#39636) >> > - Add Scarf based telemetry (#39510, #41318) >> > - Add dag re-parsing request endpoint (#39138) >> > - Redirect to new DAGRun after trigger from Grid view (#39569) >> > - Display ``endDate`` in task instance tooltip. (#39547) >> > - Implement ``accessors`` to read dataset events defined as inlet >> (#39367, >> > #39893) >> > - Add color to log lines in UI for error and warnings based on keywords >> > (#39006) >> > - Add Rendered k8s pod spec tab to ti details view (#39141) >> > - Make audit log before/after filterable (#39120) >> > - Consolidate grid collapse actions to a single full screen toggle >> (#39070) >> > - Implement Metadata to emit runtime extra (#38650) >> > - Add executor field to the DB and parameter to the operators (#38474) >> > - Implement context accessor for DatasetEvent extra (#38481) >> > - Add dataset event info to dag graph (#41012) >> > - Add button to toggle datasets on/off in dag graph (#41200) >> > - Add ``run_if`` & ``skip_if`` decorators (#41116) >> > - Add dag_stats rest api endpoint (#41017) >> > - Add listeners for Dag import errors (#39739) >> > - Allowing DateTimeSensorAsync, FileSensor and TimeSensorAsync to start >> > execution from trigger during dynamic task mapping (#41182) >> > >> > >> > *Improvements* >> > >> > - Allow set Dag Run resource into Dag Level permission: extends Dag's >> > access_control feature to allow Dag Run resource permissions. (#40703) >> > - Improve security and error handling for the internal API (#40999) >> > - Datasets UI Improvements (#40871) >> > - Change DAG Audit log tab to Event Log (#40967) >> > - Make standalone dag file processor works in DB isolation mode (#40916) >> > - Show only the source on the consumer DAG page and only triggered DAG >> run >> > in the producer DAG page (#41300) >> > - Update metrics names to allow multiple executors to report metrics >> > (#40778) >> > - Format DAG run count (#39684) >> > - Update styles for ``renderedjson`` component (#40964) >> > - Improve ATTRIBUTE_REMOVED sentinel to use class and more context >> (#40920) >> > - Make XCom display as react json (#40640) >> > - Replace usages of task context logger with the log table (#40867) >> > - Rollback for all retry exceptions (#40882) (#40883) >> > - Support rendering ObjectStoragePath value (#40638) >> > - Add try_number and map_index as params for log event endpoint (#40845) >> > - Rotate fernet key in batches to limit memory usage (#40786) >> > - Add gauge metric for 'last_num_of_db_queries' parameter (#40833) >> > - Set parallelism log messages to warning level for better visibility >> > (#39298) >> > - Add error handling for encoding the dag runs (#40222) >> > - Use params instead of dag_run.conf in example DAG (#40759) >> > - Load Example Plugins with Example DAGs (#39999) >> > - Stop deferring TimeDeltaSensorAsync task when the target_dttm is in >> the >> > past (#40719) >> > - Send important executor logs to task logs (#40468) >> > - Open external links in new tabs (#40635) >> > - Attempt to add ReactJSON view to rendered templates (#40639) >> > - Speeding up regex match time for custom warnings (#40513) >> > - Refactor DAG.dataset_triggers into the timetable class (#39321) >> > - add next_kwargs to StartTriggerArgs (#40376) >> > - Improve UI error handling (#40350) >> > - Remove double warning in CLI when config value is deprecated (#40319) >> > - Implement XComArg concat() (#40172) >> > - Added ``get_extra_dejson`` method with nested parameter which allows >> you >> > to specify if you want the nested json as string to be also deserialized >> > (#39811) >> > - Add executor field to the task instance API (#40034) >> > - Support checking for db path absoluteness on Windows (#40069) >> > - Introduce StartTriggerArgs and prevent start trigger initialization in >> > scheduler (#39585) >> > - Add task documentation to details tab in grid view (#39899) >> > - Allow executors to be specified with only the class name of the >> Executor >> > (#40131) >> > - Remove obsolete conditional logic related to try_number (#40104) >> > - Allow Task Group Ids to be passed as branches in BranchMixIn (#38883) >> > - Javascript connection form will apply CodeMirror to all textarea's >> > dynamically (#39812) >> > - Determine needs_expansion at time of serialization (#39604) >> > - Add indexes on dag_id column in referencing tables to speed up >> deletion >> > of dag records (#39638) >> > - Add task failed dependencies to details page (#38449) >> > - Remove webserver try_number adjustment (#39623) >> > - Implement slicing in lazy sequence (#39483) >> > - Unify lazy db sequence implementations (#39426) >> > - Add ``__getattr__`` to task decorator stub (#39425) >> > - Allow passing labels to FAB Views registered via Plugins (#39444) >> > - Simpler error message when trying to offline migrate with sqlite >> (#39441) >> > - Add soft_fail to TriggerDagRunOperator (#39173) >> > - Rename "dataset event" in context to use "outlet" (#39397) >> > - Resolve ``RemovedIn20Warning`` in ``airflow task`` command (#39244) >> > - Determine fail_stop on client side when db isolated (#39258) >> > - Refactor cloudpickle support in Python operators/decorators (#39270) >> > - Update trigger kwargs migration to specify existing_nullable (#39361) >> > - Allowing tasks to start execution directly from triggerer without >> going >> > to worker (#38674) >> > - Better ``db migrate`` error messages (#39268) >> > - Add stacklevel into the ``suppress_and_warn`` warning (#39263) >> > - Support searching by dag_display_name (#39008) >> > - Allow sort by on all fields in MappedInstances.tsx (#38090) >> > - Expose count of scheduled tasks in metrics (#38899) >> > - Use ``declarative_base`` from ``sqlalchemy.orm`` instead of >> > ``sqlalchemy.ext.declarative`` (#39134) >> > - Add example DAG to demonstrate emitting approaches (#38821) >> > - Give ``on_task_instance_failed`` access to the error that caused the >> > failure (#38155) >> > - Simplify dataset serialization (#38694) >> > - Add heartbeat recovery message to jobs (#34457) >> > - Remove select_column option in TaskInstance.get_task_instance (#38571) >> > - Don't create session in get_dag if not reading dags from database >> > (#38553) >> > - Add a migration script for encrypted trigger kwargs (#38358) >> > - Implement render_templates on TaskInstancePydantic (#38559) >> > - Handle optional session in _refresh_from_db (#38572) >> > - Make type annotation less confusing in task_command.py (#38561) >> > - Use fetch_dagrun directly to avoid session creation (#38557) >> > - Added ``output_processor`` parameter to ``BashProcessor`` (#40843) >> > - Improve serialization for Database Isolation Mode (#41239) >> > - Only orphan non-orphaned Datasets (#40806) >> > - Adjust gantt width based on task history dates (#41192) >> > - Enable scrolling on legend with high number of elements. (#41187) >> > >> > *Bug Fixes* >> > >> > - Bugfix for get_parsing_context() when ran with LocalExecutor (#40738) >> > - Validating provider documentation urls before displaying in views >> > (#40933) >> > - Move import to make PythonOperator working on Windows (#40424) >> > - Fix dataset_with_extra_from_classic_operator example DAG (#40747) >> > - Call listener on_task_instance_failed() after ti state is changed >> > (#41053) >> > - Add ``never_fail`` in BaseSensor (#40915) >> > - Fix tasks API endpoint when DAG doesn't have ``start_date`` (#40878) >> > - Fix and adjust URL generation for UI grid and older runs (#40764) >> > - Rotate fernet key optimization (#40758) >> > - Fix class instance vs. class type in >> > validate_database_executor_compatibility() call (#40626) >> > - Clean up dark mode (#40466) >> > - Validate expected types for args for DAG, BaseOperator and TaskGroup >> > (#40269) >> > - Exponential Backoff Not Functioning in BaseSensorOperator Reschedule >> > Mode (#39823) >> > - local task job: add timeout, to not kill on_task_instance_success >> > listener prematurely (#39890) >> > - Move Post Execution Log Grouping behind Exception Print (#40146) >> > - Fix triggerer race condition in HA setting (#38666) >> > - Pass triggered or existing DAG Run logical date to DagStateTrigger >> > (#39960) >> > - Passing ``external_task_group_id`` to ``WorkflowTrigger`` (#39617) >> > - ECS Executor: Set tasks to RUNNING state once active (#39212) >> > - Only heartbeat if necessary in backfill loop (#39399) >> > - Fix trigger kwarg encryption migration (#39246) >> > - Fix decryption of trigger kwargs when downgrading. (#38743) >> > - Fix wrong link in TriggeredDagRuns (#41166) >> > - Pass MapIndex to LogLink component for external log systems (#41125) >> > - Add NonCachingRotatingFileHandler for worker task (#41064) >> > - Add argument include_xcom in method resolve an optional value (#41062) >> > - Sanitizing file names in example_bash_decorator DAG (#40949) >> > - Show dataset aliases in dependency graphs (#41128) >> > - Render Dataset Conditions in DAG Graph view (#41137) >> > - Add task duration plot across dagruns (#40755) >> > - Add start execution from trigger support for existing core sensors >> > (#41021) >> > - add example dag for dataset_alias (#41037) >> > - Add dataset alias unique constraint and remove wrong dataset alias >> > removing logic (#41097) >> > - Set "has_outlet_datasets" to true if "dataset alias" exists (#41091) >> > - Make HookLineageCollector group datasets by (#41034) >> > - Enhance start_trigger_args serialization (#40993) >> > - Refactor ``BaseSensorOperator`` introduce ``skip_policy`` parameter >> > (#40924) >> > - Fix viewing logs from triggerer when task is deferred (#41272) >> > - Refactor how triggered dag run url is replaced (#41259) >> > - Added support for additional sql alchemy session args (#41048) >> > - Allow empty list in TriggerDagRun failed_state (#41249) >> > - Clean up the exception handler when run_as_user is the airflow user >> > (#41241) >> > - Collapse docs when click and folded (#41214) >> > - Update updated_at when saving to db as session.merge does not trigger >> > on-update (#40782) >> > - Fix query count statistics when parsing DAF file (#41149) >> > - Method Resolution Order in operators without ``__init__`` (#41086) >> > - Ensure try_number incremented for empty operator (#40426) >> > >> > *Miscellaneous* >> > >> > - Remove the Experimental flag from ``OTel`` Traces (#40874) >> > - Bump packaging version to 23.0 in order to fix issue with older otel >> > (#40865) >> > - Simplify _auth_manager_is_authorized_map function (#40803) >> > - Use correct unknown executor exception in scheduler job (#40700) >> > - Add D1 ``pydocstyle`` rules to pyproject.toml (#40569) >> > - Enable enforcing ``pydocstyle`` rule D213 in ruff. (#40448, #40464) >> > - Update ``Dag.test()`` to run with an executor if desired (#40205) >> > - Update jest and babel minor versions (#40203) >> > - Refactor BashOperator and Bash decorator for consistency and >> simplicity >> > (#39871) >> > - Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors >> > (#38778) >> > - ruff version bump 0.4.5 (#39849) >> > - Bump ``pytest`` to 8.0+ (#39450) >> > - Remove stale comment about TI index (#39470) >> > - Configure ``back_populates`` between >> ``DagScheduleDatasetReference.dag`` >> > and ``DagModel.schedule_dataset_references`` (#39392) >> > - Remove deprecation warnings in endpoints.py (#39389) >> > - Fix SQLA deprecations in Airflow core (#39211) >> > - Use class-bound attribute directly in SA (#39198, #39195) >> > - Fix stacklevel for TaskContextLogger (#39142) >> > - Capture warnings during collect DAGs (#39109) >> > - Resolve ``B028`` (no-explicit-stacklevel) in core (#39123) >> > - Rename model ``ImportError`` to ``ParseImportError`` for avoid >> shadowing >> > with builtin exception (#39116) >> > - Add option to support cloudpickle in PythonVenv/External Operator >> > (#38531) >> > - Suppress ``SubDagOperator`` examples warnings (#39057) >> > - Add log for running callback (#38892) >> > - Use ``model_dump`` instead of ``dict`` for serialize Pydantic V2 model >> > (#38933) >> > - Widen cheat sheet column to avoid wrapping commands (#38888) >> > - Update hatchling to latest version (1.22.5) (#38780) >> > - bump uv to 0.1.29 (#38758) >> > - Add missing serializations found during provider tests fixing (#41252) >> > - Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) >> > - Improve typing for allowed/failed_states in TriggerDagRunOperator >> > (#39855) >> > >> > *Doc Only Changes* >> > >> > - Add ``filesystems`` and ``dataset-uris`` to "how to create your own >> > provider" page (#40801) >> > - Fix (TM) to (R) in Airflow repository (#40783) >> > - Set ``otel_on`` to True in example airflow.cfg (#40712) >> > - Add warning for _AIRFLOW_PATCH_GEVENT (#40677) >> > - Update multi-team diagram proposal after Airflow 3 discussions >> (#40671) >> > - Add stronger warning that MSSQL is not supported and no longer >> > functional (#40565) >> > - Fix misleading mac menu structure in howto (#40440) >> > - Update k8s supported version in docs (#39878) >> > - Add compatibility note for Listeners (#39544) >> > - Update edge label image in documentation example with the new graph >> view >> > (#38802) >> > - Update UI doc screenshots (#38680) >> > - Add section "Manipulating queued dataset events through REST API" >> > (#41022) >> > - Add information about lack of security guarantees for docker compose >> > (#41072) >> > - Add links to example dags in use params section (#41031) >> > - Change ``task_id`` from ``send_email`` to ``send_email_notification`` >> in >> > ``taskflow.rst`` (#41060) >> > - Remove unnecessary nginx redirect rule from reverse proxy >> documentation >> > (#38953) >> > >> > Cheers, >> > Utkarsh Sharma >> > >> >
