This is an automated email from the ASF dual-hosted git repository.
mobuchowski pushed a change to branch listener-task-timeout
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 32e46dde86 local task job: add timeout, to not kill
on_task_instance_success listener prematurely
add 11f219abff Prepare docs 4th wave May 2024 (#39934)
add 032d27640b Refresh properties on KubernetesPodOperator on token
expiration also when logging (#39789)
add 4127ce4c2d Add `drill` integration tests (#39921)
add b7b6d1426f Resolve sshhook deprecations in tests (#39907)
add ddcc1b3a00 Fix: remove process_func from templated_fields (#39948)
add eb7826f63f Home: filter running/failed and active/paused dags (#39701)
add 93ba263732 AIP-44 Optimize Pydantic Class Serialization Mapping
(#39943)
add b8a83b2293 Workaround new yandexcloud breaking dataproc integration
(#39964)
add 5aa43e2a03 Update Databricks feature checklist comparison (#39965)
add 683dfb6f08 Resolve baseoperatormeta deprecations in tests (#39963)
add fa0aaf694e Resolving EMR notebook deprecated warnings (#39829)
add fcd1a26a9a Allow user-specified object attributes to be used in
check_fn for S3KeySensor (#39950)
add 78523fdbf1 Adding Amazon Glue Data Quality Service (#39923)
add 8173693a70 Remove upper-binding in yandex after dataproc issue is
fixed (#39974)
add c8c97b5067 Fix Mark Instance state buttons stay disabled if user lacks
permission (#37451). (#38732)
add 4d5591a509 Provide extra tip on labeling DynamicTaskMapping (#39977)
add 936c3892af Resolve triggerer job logging deprecations in tests (#39962)
add 5137aef517 Resolve timestamp deprecations in tests (#39978)
add 93e6f0070a Resolve skipmixin deprecations in tests (#39971)
add b167e3dbdd Resolve weaviate deprecations in tests (#39928)
add d93f965d84 Resolve backfill job deprecations in tests (#39961)
add 53e6739e67 Limit yandex provider to avoid mypy errors (#39990)
add 68fe455b6f Resolve taskinstance deprecations in tests (#39988)
add 3af562ff53 resolving conflicts (#39734)
add 44800b2fde Remove unused parameter from glboal_constants.py (#39984)
add ce5cd52635 Change regular expression to exclude double quote and
newline (#39991)
add 21e6a49db5 resolving conflicts (#39728)
add 30c82ee066 Resolve tableau deprecations in tests (#39925)
add 5bd07aadd6 Resolve hashicorp deprecations in tests (#39949)
add d872f7ed93 Update integration tests docs and add "Writing Integration
Tests" guide (#39986)
add 2bdaf44841 fix sub-heading in 07_local_virtualenv.rst (#39995)
add 933d80c1aa Update dataproc.rst (#39395)
add e3e450ede2 Javascript connection form will apply CodeMirror to all
textarea's dynamically (#39812)
add 357988831a Deduplicate model name in SageMakerTransformOperator
(#39956)
add bf7d5387e6 Remove rawwar from triager (#40002)
add 2ee4568010 Resolve stats deprecations in tests (#39957)
add ae648e6884 Pass triggered or existing DAG Run logical date to
DagStateTrigger (#39960)
add 7ceb3d6778 Remove deprecated docker compose version field (#40000)
add eaeb170b24 Resolve weekday operator deprecations in tests (#40004)
add 7cae13084c Fix Todo remove hybrid property hack (#39765)
add 8a9699161f Remove colon from notes header to make headlines in UI
consistent (#40001)
add c2a57a79c5 Uuups, correct bracket in Bosch name list (#39998)
add 385d79bfd7 Added ordering key option for PubSubPublishMessageOperator
GCP Operator (#39955)
add 57b12fe0ce Resolve dag serialization deprecations in tests (#40006)
add 3d97474a49 Add Shahar to triage team (#40007)
add 38e003c2c9 Warn on mini scheduler failures instead of debug (#39760)
add 1436ca472f Update providers metadata 2024-05-30 (#40021)
add bb06c56403 Allow Task Group Ids to be passed as branches in
BranchMixIn (#38883)
add 9b67a5ea33 Resolve trigger dagrun operator deprecations in tests
(#40003)
add 1bf847914e Enable templating in extraContainers and
extraInitContainers (#38507)
add da3a77a3d5 Fix triggerer race condition in HA setting (#38666)
add 981ba8f005 Fixes KubernetesPodTrigger failing running pods with
timeout (#40019)
add 354863690f Add listeners for Dag import errors (#39739)
add 651a6d6a68 standardizes template fields for `BaseSQLOperator` and adds
`database` as a templated field (#39826)
add b5bb039811 Fix bug that makes `AirflowSecurityManagerV2` leave
transactions in the `idle in transaction` state (#39935)
add 19c145c9ef Resolve aws emr deprecations in tests (#40020)
add 4849fefb50 Resolve aws provider deprecations in tests (#40026)
add dbb28d801c Resolve aws ecs deprecations in tests (#40016)
add 8a32b940ce Bedrock system test adjustment (#40032)
add 8b99ad0fbe Document `extraContainers` and `extraInitContainers` that
are templated (#40033)
add a61e789b9b Resolve common providers deprecations in tests (#40036)
add c0f27094ab iMPlement per-provider tests with lowest-direct dependency
resolution (#39946)
add 93e5800965 Remove explicitly specified folders when they are already
ignored (#40037)
add f4c90f71b9 Fixing exception types to include TypeError, which is what
is raised in (#40012)
add 32fd29d80d Fix mocking in kerberos integration test (#40042)
add 68bd42a7ff Add DatabricksTaskOperator (#40013)
add 430c4fa120 local task job: add timeout, to not kill
on_task_instance_success listener prematurely
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (32e46dde86)
\
N -- N -- N refs/heads/listener-task-timeout (430c4fa120)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 2 +-
.github/workflows/ci.yml | 5 +
.github/workflows/run-unit-tests.yml | 12 +
.github/workflows/special-tests.yml | 31 ++
Dockerfile.ci | 37 +-
INTHEWILD.md | 2 +-
airflow/dag_processing/processor.py | 7 +
airflow/jobs/local_task_job_runner.py | 8 +-
airflow/jobs/triggerer_job_runner.py | 14 +
airflow/listeners/listener.py | 3 +-
.../listeners/spec/{dataset.py => importerrors.py} | 17 +-
airflow/models/dag.py | 8 +-
airflow/models/skipmixin.py | 5 +-
airflow/models/taskinstance.py | 10 +-
airflow/operators/branch.py | 39 +-
airflow/operators/datetime.py | 8 +-
airflow/operators/python.py | 26 +-
airflow/operators/trigger_dagrun.py | 6 +-
airflow/operators/weekday.py | 6 +-
airflow/providers/amazon/CHANGELOG.rst | 14 +
airflow/providers/amazon/__init__.py | 2 +-
airflow/providers/amazon/aws/hooks/glue.py | 100 +++++
airflow/providers/amazon/aws/operators/emr.py | 68 +--
airflow/providers/amazon/aws/operators/glue.py | 264 +++++++++++-
.../providers/amazon/aws/operators/sagemaker.py | 97 ++++-
airflow/providers/amazon/aws/sensors/glue.py | 139 ++++++-
airflow/providers/amazon/aws/sensors/s3.py | 40 +-
.../amazon/aws/transfers/dynamodb_to_s3.py | 1 -
airflow/providers/amazon/aws/triggers/glue.py | 43 +-
airflow/providers/amazon/aws/waiters/glue.json | 49 +++
airflow/providers/amazon/provider.yaml | 19 +-
airflow/providers/apache/flink/provider.yaml | 2 +-
airflow/providers/apache/hdfs/provider.yaml | 6 +-
airflow/providers/apache/hive/provider.yaml | 7 +-
airflow/providers/apache/kafka/provider.yaml | 4 +-
airflow/providers/apache/kylin/provider.yaml | 2 +-
airflow/providers/apache/spark/provider.yaml | 2 +-
.../providers/celery/executors/celery_executor.py | 2 +-
airflow/providers/cloudant/provider.yaml | 2 +-
airflow/providers/cncf/kubernetes/CHANGELOG.rst | 4 +
.../cncf/kubernetes/cli/kubernetes_command.py | 4 +-
airflow/providers/cncf/kubernetes/operators/pod.py | 24 +-
airflow/providers/cncf/kubernetes/provider.yaml | 2 +-
airflow/providers/cncf/kubernetes/triggers/pod.py | 10 +-
airflow/providers/common/sql/operators/sql.py | 29 +-
airflow/providers/common/sql/operators/sql.pyi | 1 +
airflow/providers/common/sql/provider.yaml | 4 +-
.../providers/databricks/operators/databricks.py | 460 ++++++++++++++-------
airflow/providers/databricks/provider.yaml | 4 +
airflow/providers/exasol/provider.yaml | 4 +-
airflow/providers/fab/provider.yaml | 2 +-
airflow/providers/facebook/provider.yaml | 2 +-
airflow/providers/github/provider.yaml | 4 +-
airflow/providers/google/cloud/hooks/dataflow.py | 2 +-
airflow/providers/google/cloud/hooks/pubsub.py | 11 +-
airflow/providers/google/cloud/operators/pubsub.py | 18 +
airflow/providers/google/provider.yaml | 24 +-
airflow/providers/grpc/provider.yaml | 2 +-
airflow/providers/microsoft/azure/provider.yaml | 6 +-
airflow/providers/microsoft/mssql/provider.yaml | 2 +-
airflow/providers/mongo/provider.yaml | 4 +-
airflow/providers/mysql/provider.yaml | 2 +-
airflow/providers/odbc/provider.yaml | 2 +-
airflow/providers/oracle/provider.yaml | 2 +-
airflow/providers/pagerduty/provider.yaml | 2 +-
airflow/providers/papermill/provider.yaml | 3 +
airflow/providers/pinecone/provider.yaml | 2 +-
airflow/providers/postgres/provider.yaml | 2 +-
airflow/providers/presto/provider.yaml | 4 +-
airflow/providers/salesforce/provider.yaml | 4 +-
airflow/providers/sftp/provider.yaml | 2 +-
airflow/providers/snowflake/provider.yaml | 3 +-
airflow/providers/ssh/provider.yaml | 2 +-
airflow/providers/tableau/provider.yaml | 2 +-
airflow/providers/teradata/CHANGELOG.rst | 16 +
airflow/providers/teradata/__init__.py | 2 +-
airflow/providers/teradata/provider.yaml | 3 +-
airflow/providers/trino/provider.yaml | 4 +-
airflow/providers/vertica/provider.yaml | 2 +-
airflow/providers/weaviate/provider.yaml | 4 +-
airflow/providers/yandex/provider.yaml | 7 +-
airflow/serialization/serialized_objects.py | 17 +-
airflow/www/security_manager.py | 10 +-
airflow/www/static/js/connection_form.js | 36 +-
.../www/static/js/dag/details/NotesAccordion.tsx | 2 +-
.../taskInstance/taskActions/ClearInstance.tsx | 5 +-
.../taskInstance/taskActions/MarkInstanceAs.tsx | 5 +-
airflow/www/templates/airflow/dags.html | 18 +-
airflow/www/templates/airflow/grid.html | 1 +
airflow/www/views.py | 91 ++--
chart/files/pod-template-file.kubernetes-helm-yaml | 4 +-
.../dag-processor/dag-processor-deployment.yaml | 2 +-
.../templates/scheduler/scheduler-deployment.yaml | 2 +-
.../templates/triggerer/triggerer-deployment.yaml | 2 +-
.../templates/webserver/webserver-deployment.yaml | 2 +-
chart/templates/workers/worker-deployment.yaml | 2 +-
chart/values.schema.json | 20 +-
chart/values.yaml | 20 +-
contributing-docs/07_local_virtualenv.rst | 6 +-
contributing-docs/testing/integration_tests.rst | 141 ++++++-
contributing-docs/testing/unit_tests.rst | 125 ++++++
dev/breeze/doc/images/output-commands.svg | 4 +-
dev/breeze/doc/images/output_shell.svg | 236 ++++++-----
dev/breeze/doc/images/output_shell.txt | 2 +-
dev/breeze/doc/images/output_start-airflow.svg | 102 ++---
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
dev/breeze/doc/images/output_testing_db-tests.svg | 96 +++--
dev/breeze/doc/images/output_testing_db-tests.txt | 2 +-
.../images/output_testing_integration-tests.svg | 80 ++--
.../images/output_testing_integration-tests.txt | 2 +-
.../doc/images/output_testing_non-db-tests.svg | 96 +++--
.../doc/images/output_testing_non-db-tests.txt | 2 +-
dev/breeze/doc/images/output_testing_tests.svg | 174 ++++----
dev/breeze/doc/images/output_testing_tests.txt | 2 +-
.../src/airflow_breeze/commands/common_options.py | 7 +
.../airflow_breeze/commands/developer_commands.py | 10 +
.../commands/developer_commands_config.py | 2 +
.../airflow_breeze/commands/testing_commands.py | 9 +
.../commands/testing_commands_config.py | 3 +
dev/breeze/src/airflow_breeze/global_constants.py | 7 +-
.../src/airflow_breeze/params/shell_params.py | 4 +
.../templates/pyproject_TEMPLATE.toml.jinja2 | 2 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 5 +-
dev/breeze/src/airflow_breeze/utils/parallel.py | 2 +-
.../src/airflow_breeze/utils/selective_checks.py | 31 +-
dev/breeze/tests/test_packages.py | 10 +-
dev/breeze/tests/test_selective_checks.py | 20 +
docs/apache-airflow-providers-amazon/commits.rst | 16 +-
docs/apache-airflow-providers-amazon/index.rst | 6 +-
.../operators/glue.rst | 44 ++
.../commits.rst | 6 +-
.../operators/{notebook.rst => task.rst} | 28 +-
.../operators/workflow.rst | 4 +-
.../operators/cloud/dataproc.rst | 20 +-
.../operators.rst | 31 +-
docs/apache-airflow-providers-teradata/commits.rst | 14 +-
docs/apache-airflow-providers-teradata/index.rst | 20 +-
.../operators/trino.rst | 11 +-
.../administration-and-deployment/listeners.rst | 11 +-
.../dynamic-task-mapping.rst | 2 +-
docs/spelling_wordlist.txt | 1 +
generated/provider_dependencies.json | 125 +++---
generated/provider_metadata.json | 8 +
hatch_build.py | 41 +-
helm_tests/airflow_aux/test_create_user_job.py | 14 +
.../airflow_aux/test_migrate_database_job.py | 14 +
helm_tests/airflow_aux/test_pod_template_file.py | 30 ++
helm_tests/airflow_core/test_dag_processor.py | 30 ++
helm_tests/airflow_core/test_scheduler.py | 29 ++
helm_tests/airflow_core/test_triggerer.py | 28 ++
helm_tests/airflow_core/test_worker.py | 29 ++
helm_tests/webserver/test_webserver.py | 29 ++
.../ci/docker-compose/drill/drill-override.conf | 15 +-
.../ci/docker-compose/integration-cassandra.yml | 1 -
scripts/ci/docker-compose/integration-celery.yml | 1 -
...integration-mongo.yml => integration-drill.yml} | 30 +-
scripts/ci/docker-compose/integration-kafka.yml | 1 -
scripts/ci/docker-compose/integration-kerberos.yml | 1 -
scripts/ci/docker-compose/integration-mongo.yml | 1 -
scripts/ci/docker-compose/integration-mssql.yml | 1 -
.../ci/docker-compose/integration-openlineage.yml | 1 -
scripts/ci/docker-compose/integration-otel.yml | 1 -
scripts/ci/docker-compose/integration-pinot.yml | 1 -
scripts/ci/docker-compose/integration-qdrant.yml | 1 -
scripts/ci/docker-compose/integration-statsd.yml | 1 -
scripts/ci/docker-compose/integration-trino.yml | 1 -
scripts/docker/entrypoint_ci.sh | 37 +-
scripts/in_container/check_environment.sh | 4 +
scripts/in_container/run_ci_tests.sh | 10 +
tests/always/test_example_dags.py | 4 -
tests/core/test_stats.py | 25 +-
tests/dags/test_mark_state.py | 2 +
tests/deprecations_ignore.yml | 154 -------
.../auth/backend/test_kerberos_auth.py | 5 +-
.../providers/apache/drill/hooks}/__init__.py | 0
.../providers/apache/drill/hooks/test_drill.py | 72 ++++
.../providers/apache/drill/operators/test_drill.py | 63 ---
tests/jobs/test_backfill_job.py | 56 +--
tests/jobs/test_local_task_job.py | 3 +-
tests/jobs/test_triggerer_job.py | 80 ++++
tests/jobs/test_triggerer_job_logging.py | 3 -
...ag_listener.py => dag_import_error_listener.py} | 35 +-
tests/listeners/test_dag_import_error_listener.py | 166 ++++++++
tests/models/test_baseoperatormeta.py | 3 +-
tests/models/test_skipmixin.py | 44 +-
tests/models/test_taskinstance.py | 46 ++-
tests/models/test_timestamp.py | 4 +-
tests/operators/test_branch_operator.py | 42 ++
tests/operators/test_python.py | 16 +-
tests/operators/test_trigger_dagrun.py | 102 ++++-
tests/operators/test_weekday.py | 4 +
tests/providers/amazon/aws/hooks/test_glue.py | 142 ++++++-
tests/providers/amazon/aws/hooks/test_sagemaker.py | 28 +-
tests/providers/amazon/aws/operators/test_ecs.py | 10 +-
.../aws/operators/test_emr_notebook_execution.py | 8 +-
.../amazon/aws/operators/test_emr_serverless.py | 164 +++++---
tests/providers/amazon/aws/operators/test_glue.py | 246 ++++++++++-
.../amazon/aws/operators/test_sagemaker_base.py | 92 ++++-
.../aws/operators/test_sagemaker_transform.py | 113 ++++-
tests/providers/amazon/aws/sensors/test_ecs.py | 6 +-
.../amazon/aws/sensors/test_glue_data_quality.py | 182 ++++++++
tests/providers/amazon/aws/sensors/test_s3.py | 141 +++++++
tests/providers/amazon/aws/triggers/test_glue.py | 40 +-
.../amazon/aws/utils/test_connection_wrapper.py | 13 +-
tests/providers/amazon/aws/waiters/test_glue.py | 72 ++++
.../apache/spark/operators/test_spark_sql.py | 2 +-
.../cncf/kubernetes/operators/test_pod.py | 34 +-
.../providers/cncf/kubernetes/triggers/test_pod.py | 37 ++
tests/providers/common/sql/hooks/test_dbapi.py | 7 +-
tests/providers/common/sql/operators/test_sql.py | 41 +-
.../databricks/operators/test_databricks.py | 96 ++++-
.../providers/google/cloud/hooks/test_dataflow.py | 17 +
tests/providers/google/cloud/hooks/test_pubsub.py | 9 +-
.../google/cloud/operators/test_pubsub.py | 18 +
tests/providers/hashicorp/secrets/test_vault.py | 33 +-
tests/providers/ssh/hooks/test_ssh.py | 120 +++---
tests/providers/tableau/hooks/test_tableau.py | 11 +-
.../providers/weaviate/operators/test_weaviate.py | 30 +-
tests/serialization/test_dag_serialization.py | 36 +-
tests/serialization/test_pydantic_models.py | 4 +-
.../aws/example_bedrock_retrieve_and_generate.py | 5 +
...mple_athena.py => example_glue_data_quality.py} | 134 +++---
.../providers/databricks/example_databricks.py | 37 ++
.../databricks/example_databricks_workflow.py | 41 +-
.../providers/microsoft/mssql/example_mssql.py | 26 +-
tests/system/providers/trino/example_trino.py | 19 +-
tests/www/test_security_manager.py | 31 ++
tests/www/views/test_views_base.py | 2 +-
tests/www/views/test_views_home.py | 22 +-
229 files changed, 5278 insertions(+), 1644 deletions(-)
copy airflow/listeners/spec/{dataset.py => importerrors.py} (75%)
copy docs/apache-airflow-providers-databricks/operators/{notebook.rst =>
task.rst} (54%)
copy airflow/providers/fab/auth_manager/security_manager/constants.py =>
scripts/ci/docker-compose/drill/drill-override.conf (84%)
copy scripts/ci/docker-compose/{integration-mongo.yml =>
integration-drill.yml} (69%)
copy {airflow/api_connexion =>
tests/integration/providers/apache/drill/hooks}/__init__.py (100%)
create mode 100644 tests/integration/providers/apache/drill/hooks/test_drill.py
delete mode 100644
tests/integration/providers/apache/drill/operators/test_drill.py
copy tests/listeners/{dag_listener.py => dag_import_error_listener.py} (60%)
create mode 100644 tests/listeners/test_dag_import_error_listener.py
create mode 100644 tests/providers/amazon/aws/sensors/test_glue_data_quality.py
create mode 100644 tests/providers/amazon/aws/waiters/test_glue.py
copy tests/system/providers/amazon/aws/{example_athena.py =>
example_glue_data_quality.py} (56%)
