This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 144eb51d6b Fix check on subclass for `typing.Union` in
`_infer_multiple_outputs` for Python 3.10+ (#36728)
new 781e917a06 Update providers metadata 2023-12-31 (#36507)
new 3d04133d4d Update description of release process to include options
for next RCs (#36506)
new 332d971c78 Clarify how to find changelog for the RC candidates of
providers (#36512)
new ba5278b75a Announce MSSQL support end in Airflow 2.9.0, add migration
script hints (#36509)
new 18c60e8598 feat: K8S 1.29 support (#36527)
new 954a20280d Add pre-commit check to check integrations table. (#36497)
new 38082a19eb Implement Google Analytics Admin (GA4) operators (#36276)
new 950b08f425 Add support for openlineage to AFS and common.io (#36410)
new b8e4dcde20 Optimise getting changelogs in airflow-github (#36548)
new 587b0a2e37 Tracking airflow-github changelog activities using progress
bar (#36610)
new 4dee09153d Upgrade to latest versions of `pip` and `pipx` in CI
runners (#36646)
new b4205c00b3 Run mypy checks for full packages in CI (#36638)
new 8617a63261 Add Airflow logo to the README (#36645)
new e0fe4bf7bc Make `graphviz` dependency optional (#36647)
new 188d121d95 Upgrade to latest ruff and remove ISC001 warning from
output (#36649)
new b2f3ade873 Bump min version for grpcio-status in spark provider
(#36662)
new b803e3c36a Bump Cohere min version to limit backtracking on upgrade
(#36666)
new 93018f4c2f Bump min version of mysql-connector-python (#36668)
new def648156b Bump min provider version for papermill (#36669)
new bd7c6c8879 Bump min telegram-bot version (#36670)
new 05ebe7ad49 Add version check for k8s setup venv command (#36673)
new f3eeeca20d Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
new 64834807dc Explicitly passing `raise_on_deleted_version=True` to
`read_secret_version` in Hashicorp operator (#36532)
new c682bc4182 Update providers metadata 2024-01-10 (#36718)
new dfa98fc200 Standardize airflow build process and switch to Hatchling
build backend (#36537)
new 9a5f702e8d Add support for reproducible build date epoch for Airflow
releases (#36726)
new 18cbdf6364 Add support of Pendulum 3 (#36281)
new 2527fb0e09 Update Airflow release process to include reproducible
tarballs (#36744)
The 28 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dockerignore | 3 -
.github/actions/build-prod-images/action.yml | 2 +-
.github/workflows/ci.yml | 189 ++-
.gitignore | 7 +-
.pre-commit-config.yaml | 110 +-
.rat-excludes | 8 +-
.../LICENSE-bootstrap.txt | 0
.../LICENSE-bootstrap3-typeahead.txt | 0
.../LICENSE-d3-shape.txt | 0
.../LICENSE-d3-tip.txt | 0
{licenses => 3rd-party-licenses}/LICENSE-d3js.txt | 0
.../LICENSE-dagre-d3.txt | 0
.../LICENSE-datatables.txt | 0
.../LICENSE-elasticmock.txt | 0
.../LICENSE-eonasdan-bootstrap-datetimepicker.txt | 0
.../LICENSE-flask-kerberos.txt | 0
{licenses => 3rd-party-licenses}/LICENSE-hue.txt | 0
.../LICENSE-jqclock.txt | 0
.../LICENSE-jquery.txt | 0
.../LICENSE-moment.txt | 0
.../LICENSE-normalize.txt | 0
.../LICENSE-pytest-capture-warnings.txt | 0
3rd-party-licenses/LICENSE-reproducible.txt | 8 +
.../LICENSE-unicodecsv.txt | 0
BREEZE.rst | 51 +-
CI.rst | 4 +-
CONTRIBUTING.rst | 146 ++-
CONTRIBUTORS_QUICK_START.rst | 5 +-
Dockerfile | 19 +-
Dockerfile.ci | 59 +-
IMAGES.rst | 2 +-
INSTALL | 342 ++++--
LOCAL_VIRTUALENV.rst | 337 +++--
MANIFEST.in | 43 -
README.md | 31 +-
STATIC_CODE_CHECKS.rst | 48 +-
TESTING.rst | 45 +-
airflow/_vendor/README.md | 2 +-
airflow/cli/cli_parser.py | 9 +-
airflow/configuration.py | 2 +-
airflow/contrib/operators/__init__.py | 189 ++-
airflow/contrib/secrets/__init__.py | 2 +-
airflow/contrib/sensors/__init__.py | 10 +-
airflow/contrib/utils/__init__.py | 4 +-
airflow/decorators/__init__.pyi | 18 +-
airflow/io/path.py | 11 +-
airflow/models/dag.py | 6 +-
airflow/models/taskreschedule.py | 3 +-
airflow/operators/latest_only.py | 2 +-
airflow/provider.yaml.schema.json | 7 +
airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst | 83 +-
airflow/providers/amazon/provider.yaml | 50 +-
airflow/providers/apache/beam/provider.yaml | 10 +-
airflow/providers/apache/druid/hooks/druid.py | 4 +-
airflow/providers/apache/hive/provider.yaml | 2 +-
airflow/providers/apache/impala/provider.yaml | 11 +-
airflow/providers/apache/spark/provider.yaml | 12 +-
airflow/providers/celery/provider.yaml | 11 +-
.../cncf/kubernetes/pod_launcher_deprecated.py | 6 +-
airflow/providers/cohere/provider.yaml | 2 +-
.../providers/common/io/operators/file_transfer.py | 37 +-
.../providers/databricks/hooks/databricks_sql.py | 4 +-
airflow/providers/databricks/provider.yaml | 17 +-
airflow/providers/exasol/hooks/exasol.py | 6 +-
airflow/providers/google/cloud/hooks/cloud_run.py | 2 +-
.../providers/google/cloud/triggers/cloud_run.py | 2 +-
.../google/cloud/utils/credentials_provider.py | 2 +-
.../providers/google/common/hooks/base_google.py | 2 +-
.../google/common/utils/id_token_credentials.py | 4 +-
.../google/marketing_platform/hooks/analytics.py | 9 +
.../marketing_platform/hooks/analytics_admin.py | 234 ++++
.../google/marketing_platform/links}/__init__.py | 0
.../marketing_platform/links/analytics_admin.py | 65 +
.../marketing_platform/operators/analytics.py | 53 +
.../operators/analytics_admin.py | 579 +++++++++
airflow/providers/google/provider.yaml | 54 +-
airflow/providers/grpc/hooks/grpc.py | 2 +-
.../hashicorp/_internal_client/vault_client.py | 10 +-
airflow/providers/hashicorp/provider.yaml | 2 +-
airflow/providers/microsoft/azure/provider.yaml | 3 +
airflow/providers/mongo/provider.yaml | 3 +
airflow/providers/mysql/provider.yaml | 12 +-
airflow/providers/openlineage/plugins/facets.py | 4 +-
airflow/providers/oracle/provider.yaml | 11 +-
airflow/providers/papermill/provider.yaml | 2 +-
airflow/providers/postgres/hooks/postgres.py | 4 +-
airflow/providers/postgres/provider.yaml | 10 +-
airflow/providers/snowflake/hooks/snowflake.py | 6 +-
airflow/providers/tabular/provider.yaml | 3 +
airflow/providers/telegram/provider.yaml | 2 +-
airflow/providers/trino/operators/trino.py | 8 +-
airflow/providers/vertica/hooks/vertica.py | 2 +-
airflow/providers_manager.py | 18 +-
airflow/reproducible_build.yaml | 2 +
airflow/sensors/base.py | 4 +-
airflow/serialization/serialized_objects.py | 9 +-
airflow/serialization/serializers/datetime.py | 14 +-
airflow/serialization/serializers/timezone.py | 7 +-
airflow/settings.py | 11 +-
airflow/timetables/_cron.py | 9 +-
airflow/timetables/trigger.py | 18 +-
airflow/utils/dot_renderer.py | 15 +-
airflow/utils/operator_helpers.py | 2 +-
airflow/utils/sqlalchemy.py | 5 +-
airflow/utils/timezone.py | 50 +-
airflow/www/webpack.config.js | 2 +-
clients/gen/common.sh | 3 +-
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 7 +-
dev/README.md | 11 +-
dev/README_RELEASE_AIRFLOW.md | 55 +-
dev/README_RELEASE_PROVIDER_PACKAGES.md | 113 +-
dev/airflow-github | 50 +-
dev/airflow_pre_installed_providers.txt | 7 +
dev/breeze/README.md | 2 +-
dev/breeze/SELECTIVE_CHECKS.md | 29 +-
.../adr/0003-bootstrapping-virtual-environment.md | 4 +-
dev/breeze/pyproject.toml | 4 +-
.../airflow_breeze/commands/ci_image_commands.py | 4 -
.../commands/ci_image_commands_config.py | 1 -
.../commands/common_image_options.py | 6 -
.../src/airflow_breeze/commands/common_options.py | 6 +
.../airflow_breeze/commands/developer_commands.py | 16 +-
.../commands/developer_commands_config.py | 3 +-
.../commands/production_image_commands.py | 4 -
.../commands/production_image_commands_config.py | 1 -
.../commands/release_candidate_command.py | 139 ++-
.../commands/release_management_commands.py | 193 ++-
.../commands/release_management_commands_config.py | 16 +-
.../airflow_breeze/commands/testing_commands.py | 6 +
.../commands/testing_commands_config.py | 3 +
dev/breeze/src/airflow_breeze/global_constants.py | 14 +-
.../src/airflow_breeze/params/build_ci_params.py | 2 +-
.../src/airflow_breeze/params/build_prod_params.py | 1 -
.../airflow_breeze/params/common_build_params.py | 1 -
.../src/airflow_breeze/params/shell_params.py | 4 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 6 +-
.../prepare_providers/provider_documentation.py | 2 +-
.../prepare_providers/provider_packages.py | 3 +-
.../airflow_breeze/utils/docker_command_utils.py | 5 +-
.../src/airflow_breeze/utils/kubernetes_utils.py | 29 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 12 +-
.../src/airflow_breeze/utils/reproducible.py | 135 ++
.../src/airflow_breeze/utils/selective_checks.py | 70 +-
dev/breeze/tests/test_selective_checks.py | 254 +++-
dev/hatch_build.py | 176 +++
.../prod_image_installed_providers.txt | 1 +
dev/refresh_images.sh | 2 +-
dev/sign.sh | 4 +-
docker_tests/docker_tests_utils.py | 3 +-
docker_tests/test_prod_image.py | 37 +-
.../apache-airflow-providers-apache-hive/index.rst | 2 +-
.../operators/marketing_platform/analytics.rst | 5 +
.../marketing_platform/analytics_admin.rst | 151 +++
.../howto/create-custom-providers.rst | 36 +-
.../modules_management.rst | 54 +-
.../apache-airflow/core-concepts/objectstorage.rst | 8 +-
docs/apache-airflow/extra-packages-ref.rst | 202 +--
docs/apache-airflow/howto/set-up-database.rst | 30 +-
.../installation/installing-from-pypi.rst | 6 +-
docs/apache-airflow/installation/prerequisites.rst | 2 +-
docs/docker-stack/build-arg-ref.rst | 17 +-
docs/spelling_wordlist.txt | 3 +
generated/PYPI_README.md | 29 +-
generated/provider_dependencies.json | 125 +-
generated/provider_metadata.json | 84 ++
images/breeze/output_ci-image_build.svg | 118 +-
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_k8s_configure-cluster.svg | 4 +-
images/breeze/output_k8s_configure-cluster.txt | 2 +-
images/breeze/output_k8s_create-cluster.svg | 4 +-
images/breeze/output_k8s_create-cluster.txt | 2 +-
images/breeze/output_k8s_delete-cluster.svg | 2 +-
images/breeze/output_k8s_delete-cluster.txt | 2 +-
images/breeze/output_k8s_deploy-airflow.svg | 4 +-
images/breeze/output_k8s_deploy-airflow.txt | 2 +-
images/breeze/output_k8s_k9s.svg | 2 +-
images/breeze/output_k8s_k9s.txt | 2 +-
images/breeze/output_k8s_logs.svg | 2 +-
images/breeze/output_k8s_logs.txt | 2 +-
images/breeze/output_k8s_run-complete-tests.svg | 4 +-
images/breeze/output_k8s_run-complete-tests.txt | 2 +-
images/breeze/output_k8s_shell.svg | 2 +-
images/breeze/output_k8s_shell.txt | 2 +-
images/breeze/output_k8s_status.svg | 2 +-
images/breeze/output_k8s_status.txt | 2 +-
images/breeze/output_k8s_tests.svg | 4 +-
images/breeze/output_k8s_tests.txt | 2 +-
images/breeze/output_k8s_upload-k8s-image.svg | 4 +-
images/breeze/output_k8s_upload-k8s-image.txt | 2 +-
images/breeze/output_prod-image_build.svg | 148 ++-
images/breeze/output_prod-image_build.txt | 2 +-
images/breeze/output_release-management.svg | 54 +-
images/breeze/output_release-management.txt | 2 +-
...put_release-management_generate-constraints.svg | 2 +-
...put_release-management_generate-constraints.txt | 2 +-
..._release-management_prepare-airflow-package.svg | 38 +-
..._release-management_prepare-airflow-package.txt | 2 +-
..._release-management_prepare-airflow-tarball.svg | 106 ++
..._release-management_prepare-airflow-tarball.txt | 1 +
.../output_setup_check-all-params-in-groups.svg | 20 +-
.../output_setup_check-all-params-in-groups.txt | 2 +-
.../output_setup_regenerate-command-images.svg | 46 +-
.../output_setup_regenerate-command-images.txt | 2 +-
images/breeze/output_shell.svg | 44 +-
images/breeze/output_shell.txt | 2 +-
images/breeze/output_start-airflow.svg | 8 +-
images/breeze/output_start-airflow.txt | 2 +-
images/breeze/output_static-checks.svg | 56 +-
images/breeze/output_static-checks.txt | 2 +-
images/breeze/output_testing_db-tests.svg | 26 +-
images/breeze/output_testing_db-tests.txt | 2 +-
images/breeze/output_testing_non-db-tests.svg | 26 +-
images/breeze/output_testing_non-db-tests.txt | 2 +-
images/breeze/output_testing_tests.svg | 26 +-
images/breeze/output_testing_tests.txt | 2 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 5 +-
newsfragments/36281.significant.rst | 4 +
newsfragments/36537.significant.rst | 41 +
newsfragments/36647.significant.rst | 23 +
pyproject.toml | 1288 +++++++++++++++++++-
scripts/ci/docker-compose/devcontainer.env | 2 +-
.../ci/docker-compose/integration-cassandra.yml | 3 +
scripts/ci/docker-compose/integration-celery.yml | 3 +
scripts/ci/docker-compose/integration-kafka.yml | 3 +
scripts/ci/docker-compose/integration-kerberos.yml | 3 +
scripts/ci/docker-compose/integration-mongo.yml | 3 +
.../ci/docker-compose/integration-openlineage.yml | 4 +-
scripts/ci/docker-compose/integration-otel.yml | 3 +
scripts/ci/docker-compose/integration-pinot.yml | 3 +
scripts/ci/docker-compose/integration-statsd.yml | 3 +
scripts/ci/docker-compose/integration-trino.yml | 3 +
scripts/ci/docker-compose/local.yml | 9 -
scripts/ci/install_breeze.sh | 3 +-
scripts/ci/kubernetes/k8s_requirements.txt | 2 +-
scripts/ci/pre_commit/common_precommit_utils.py | 26 +-
.../pre_commit_check_extra_packages_ref.py | 97 ++
.../pre_commit_check_integrations_list.py | 202 +++
.../pre_commit_check_order_dockerfile_extras.py | 2 +-
.../pre_commit_check_order_pyproject_toml.py | 103 ++
.../ci/pre_commit/pre_commit_check_order_setup.py | 135 --
.../pre_commit_check_setup_extra_packages_ref.py | 260 ----
.../ci/pre_commit/pre_commit_compile_www_assets.py | 18 +-
scripts/ci/pre_commit/pre_commit_insert_extras.py | 91 +-
scripts/ci/pre_commit/pre_commit_mypy.py | 31 +-
.../pre_commit_ruff_format.py} | 14 +-
.../pre_commit_sort_installed_providers.py | 15 +-
.../pre_commit_update_providers_dependencies.py | 171 ++-
..._commit_update_providers_dependencies.py.md5sum | 1 +
.../pre_commit_update_source_date_epoch.py | 45 +
scripts/docker/entrypoint_ci.sh | 16 +-
scripts/docker/install_airflow.sh | 7 +-
...install_airflow_dependencies_from_branch_tip.sh | 4 +-
scripts/in_container/run_generate_constraints.py | 38 +-
.../in_container/run_prepare_airflow_packages.py | 202 +--
scripts/in_container/verify_providers.py | 11 +-
scripts/tools/initialize_virtualenv.py | 9 +-
setup.cfg | 245 ----
setup.py | 1079 ----------------
tests/api_connexion/endpoints/test_dag_endpoint.py | 12 +-
tests/api_connexion/schemas/test_dag_schema.py | 5 +-
tests/cli/commands/test_dag_command.py | 13 +-
tests/cli/commands/test_task_command.py | 10 +-
tests/io/test_path.py | 11 +-
tests/models/test_dag.py | 9 +-
tests/plugins/test_plugins_manager.py | 3 +-
.../common/io/operators/test_file_transfer.py | 24 +
.../providers/google/cloud/utils/base_gcp_mock.py | 4 +-
.../hooks/test_analytics_admin.py | 192 +++
.../google/marketing_platform/links}/__init__.py | 0
.../links/test_analytics_admin.py | 71 ++
.../operators/test_analytics_admin.py | 310 +++++
.../_internal_client/test_vault_client.py | 12 +-
tests/providers/hashicorp/hooks/test_vault.py | 6 +-
tests/providers/hashicorp/secrets/test_vault.py | 4 +-
tests/providers/openlineage/plugins/test_utils.py | 6 +-
tests/sensors/test_time_sensor.py | 19 +-
.../serialization/serializers/test_serializers.py | 152 +++
tests/serialization/test_serialized_objects.py | 4 +-
.../marketing_platform/example_analytics_admin.py | 203 +++
tests/triggers/test_temporal.py | 6 +-
tests/utils/test_timezone.py | 47 +-
281 files changed, 7891 insertions(+), 3696 deletions(-)
rename {licenses => 3rd-party-licenses}/LICENSE-bootstrap.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-bootstrap3-typeahead.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-d3-shape.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-d3-tip.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-d3js.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-dagre-d3.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-datatables.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-elasticmock.txt (100%)
rename {licenses =>
3rd-party-licenses}/LICENSE-eonasdan-bootstrap-datetimepicker.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-flask-kerberos.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-hue.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-jqclock.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-jquery.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-moment.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-normalize.txt (100%)
rename {licenses => 3rd-party-licenses}/LICENSE-pytest-capture-warnings.txt
(100%)
create mode 100644 3rd-party-licenses/LICENSE-reproducible.txt
rename {licenses => 3rd-party-licenses}/LICENSE-unicodecsv.txt (100%)
delete mode 100644 MANIFEST.in
create mode 100644
airflow/providers/google/marketing_platform/hooks/analytics_admin.py
copy airflow/{api_connexion =>
providers/google/marketing_platform/links}/__init__.py (100%)
create mode 100644
airflow/providers/google/marketing_platform/links/analytics_admin.py
create mode 100644
airflow/providers/google/marketing_platform/operators/analytics_admin.py
create mode 100644 airflow/reproducible_build.yaml
create mode 100644 dev/airflow_pre_installed_providers.txt
create mode 100644 dev/breeze/src/airflow_breeze/utils/reproducible.py
create mode 100644 dev/hatch_build.py
rename airflow/providers/installed_providers.txt =>
dev/prod_image_installed_providers.txt (74%)
create mode 100644
docs/apache-airflow-providers-google/operators/marketing_platform/analytics_admin.rst
create mode 100644
images/breeze/output_release-management_prepare-airflow-tarball.svg
create mode 100644
images/breeze/output_release-management_prepare-airflow-tarball.txt
create mode 100644 newsfragments/36281.significant.rst
create mode 100644 newsfragments/36537.significant.rst
create mode 100644 newsfragments/36647.significant.rst
create mode 100755 scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
create mode 100755 scripts/ci/pre_commit/pre_commit_check_integrations_list.py
create mode 100755
scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
delete mode 100755 scripts/ci/pre_commit/pre_commit_check_order_setup.py
delete mode 100755
scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
copy scripts/ci/{install_breeze.sh => pre_commit/pre_commit_ruff_format.py}
(71%)
create mode 100644
scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py.md5sum
create mode 100755 scripts/ci/pre_commit/pre_commit_update_source_date_epoch.py
delete mode 100644 setup.cfg
delete mode 100644 setup.py
create mode 100644
tests/providers/google/marketing_platform/hooks/test_analytics_admin.py
copy {airflow/api/auth =>
tests/providers/google/marketing_platform/links}/__init__.py (100%)
create mode 100644
tests/providers/google/marketing_platform/links/test_analytics_admin.py
create mode 100644
tests/providers/google/marketing_platform/operators/test_analytics_admin.py
create mode 100644
tests/system/providers/google/marketing_platform/example_analytics_admin.py
