This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch remove-python-3-7
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 5025551d1a Remove Python 3.7 support
add a2a9867ad1 Update callbacks documentation (errors and context) (#31116)
add 2af2ddca06 Added index on taskinstance which includes the start_date
column (#30762)
add f36587bb46 Update version in API spec (#31086)
add e68d4a7764 implement gcs_schema_object for
BigQueryCreateExternalTableOperator (#30961)
add 10dda55e8b Fix acl test on s3 copy (#31138)
add 672ee7f0e1 Ensure that we check worker for served logs if no local or
remote logs found (#31101)
add dff7e0de36 Revert "DynamoDBToS3Operator - Add feature to export table
to a point in time (#30501)" (#31139)
add b8f73768ec Add `as_dict` param to `BigQueryGetDataOperator` (#30887)
add 4c9b5fe4c1 Add deferrable param in SageMakerTransformOperator (#31063)
add 521dae534d Redirecting to the sub-section in context (#31131)
add edd7133a13 Add conditional output processing in SQL operators (#31136)
add 25e315995e Remove worrying log message about redaction from the
OpenLineage plugin (#31149)
add bc7a3ab4fc Add description on how we handle security issues in Airflow
(#30960)
add 89e0c5173d Update Adyen in INTOTHEWILD.md (#31154)
add 130b6763db Revert "Add description on how we handle security issues in
Airflow (#30960)" (#31158)
add edebfe3f2f Add docs for Oracle operators (#30979)
add 8db0adebb0 Add directly dependent providers to package installation
tests. (#31163)
add ef69f61778 Add SECURITY.rst to doc files in selective checks (#31161)
add 2b26a76803 Add sorting logic by created_date for fetching triggers
(#31151)
add 41cb63764d Fix deprecated import in Oracle example (#31166)
add b4ba6b7cb1 Use only one line for tmp dir log (#31170)
add 71c26276bc Ensure the KPO runs pod mutation hooks correctly (#31173)
add cd3fa33e82 DynamoDBToS3Operator - Add a feature to export the table to
a point in time. (#31142)
add 289065285c Fix section name reference for
api_client_retry_configuration (#31174)
add f39ccad9d8 Executor events are not always "exited" here (#30859)
add f8f47b0f83 Initialize pre-commit environment and retry if it fails
(#31164)
add c3c90eaa94 Add options to KubernetesPodOperator (#30992)
add 420a9b1581 Make sure that DAG processor job row has filed value in
`job_type` column (#31182)
add 584a9f5dae Added spinner activity while the logs load (#31165)
add 8eab2e54ee K8s hook should still work with missing default conn
(#31187)
add 8a5fe6ad7d Add retries to S3 delete_bucket (#31192)
add 86d62d3c03 Fix timestamp parse failure for k8s executor pod tailing
(#31175)
add 60536152b2 Add description on security issue handling in Airflow
(#31160)
add d4dc73416a Remove the upper version bound on pymongo (#31189)
add fc69214f03 Support for sorting DAGs by Last Run Date in the web UI
(#30935)
add 1157470bc0 Fixed typo in `PagerdutyEventsHook` docstring (#31209)
add ec2f0f12db Optimize deferred execution for
AzureDataFactoryRunPipelineOperator (#31214)
add e7cb9ca2c1 Fix pod describing on system test failure (#31191)
add 97215822f9 Improve docker-compose test usability and debuggability
(#31181)
add 9e10bf1cf0 Revert "Support for sorting DAGs by Last Run Date in the
web UI (#30935)" (#31227)
add 774a5a9038 Empty xcom result file log message more specific (#31228)
add be8f96baa2 Fix circular import (#31204)
add 249ff2f361 fix trigger rule (#31235)
add 0d962c2967 Add throttling section in EMR doc (#31224)
add 85482e86f5 With "full tests needed" label - run mssql tests on public
runners (#31247)
add 5cb8ef80a0 Validate connection IDs (#31140)
add 24532312b6 Standardize Slack Notifier (#31244)
add d1fe67184d Add `use_legacy_sql` param to `BigQueryGetDataOperator`
(#31190)
add 03b04a3d54 Resolving problems with redesigned grid veiw (#31232)
add 981afe2a4f openlineage: add extractors for python and bash operators
(#30713)
add 1395754487 Upper bind dask until they solve a side effect in their
test suite (#31259)
add 5f208ed33b Fix pymongo tests to be forward-compatible (#31257)
add 37d8d62a7e Add missing DynamoDBToS3Operator system test (#31197)
add 2b4ad8a18c Fix `url_for_asset` fallback and 404 on DAG Audit Log
(#31233)
add 761c0da723 Upper-bind elasticearch integration (#31255)
add d6051fd10a Add on_kill to EMR Serverless Job Operator (#31169)
add d59b0533e1 Fix task.branch error caused by incorrect initial parameter
(#31265)
add 1133035f79 Upgrade FAB to 4.3.1 (#31203)
add 47a9fff5c3 Fix invalid escape sequences in strings in Breeze (#31271)
add fe59f4ae21 Remove unused fixtures in slack tests (#31268)
add 6b21e4b88c Bring back detection of implicit single-line string
concatenation (#31270)
add 779af8238a Add --only-my-changes flag to breeze static-checks command
(#31269)
add a66edcbb2c `CreateBatchPredictionJobOperator` Add batch_size param for
Vertex AI BatchPredictionJob objects (#31118)
add fdc7a31aeb Add `DatabricksPartitionSensor` (#30980)
add fb6c501124 Add flag apply_gcs_prefix to S3ToGCSOperator (b/245077385)
(#31127)
add 3193857376 Add Deferrable Mode for EC2StateSensor (#31130)
add f366d955cd Fix calculation of health check threshold for SchedulerJob
(#31277)
add 9432a3f463 Better typing for Job and JobRunners (#31240)
add 3aee835885 Correct minor typo impocit->implicit in pyproject.toml
comment (#31283)
add 9a8c2044bd Remove snakebite-py3 based HDFS hooks and sensors (#31262)
add 2b6462b5a4 Make BaseJobRunner a generic on the job class (#31287)
add c605ef0ca3 Remove --all-files specification in basic checks in CI
(#31286)
add 0117246db6 Add future-compatible mongo Hook typing (#31289)
add 2e912a657c Remove Python 3.7 support
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5025551d1a)
\
N -- N -- N refs/heads/remove-python-3-7 (2e912a657c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/SECURITY.rst | 69 ++++-
.github/workflows/ci.yml | 18 +-
BREEZE.rst | 43 ++-
COMMITTERS.rst | 6 +
INTHEWILD.md | 2 +-
STATIC_CODE_CHECKS.rst | 13 +-
TESTING.rst | 78 +++++
airflow/api_connexion/openapi/v1.yaml | 2 +-
airflow/config_templates/config.yml | 12 +-
airflow/config_templates/default_airflow.cfg | 7 +-
airflow/contrib/sensors/__init__.py | 6 -
airflow/executors/kubernetes_executor.py | 12 +-
airflow/hooks/__init__.py | 4 -
airflow/hooks/subprocess.py | 2 +-
airflow/jobs/backfill_job_runner.py | 11 +-
airflow/jobs/base_job_runner.py | 16 +-
airflow/jobs/dag_processor_job_runner.py | 10 +-
airflow/jobs/job.py | 7 +-
airflow/jobs/local_task_job_runner.py | 11 +-
airflow/jobs/scheduler_job_runner.py | 41 +--
airflow/jobs/triggerer_job_runner.py | 11 +-
airflow/kubernetes/kube_client.py | 4 +-
...0125_2_7_0_add_index_to_task_instance_table.py} | 25 +-
airflow/models/taskinstance.py | 9 +
airflow/models/trigger.py | 21 +-
airflow/providers/amazon/aws/hooks/batch_client.py | 2 +-
airflow/providers/amazon/aws/hooks/ec2.py | 5 +
airflow/providers/amazon/aws/hooks/s3.py | 19 +-
airflow/providers/amazon/aws/operators/batch.py | 2 +-
airflow/providers/amazon/aws/operators/emr.py | 43 ++-
.../providers/amazon/aws/operators/sagemaker.py | 53 +++-
airflow/providers/amazon/aws/sensors/ec2.py | 27 +-
.../amazon/aws/transfers/dynamodb_to_s3.py | 3 +-
.../providers/amazon/aws/transfers/mongo_to_s3.py | 5 +-
.../amazon/aws/{sensors => triggers}/ec2.py | 68 ++---
airflow/providers/apache/hdfs/CHANGELOG.rst | 18 ++
airflow/providers/apache/hdfs/hooks/hdfs.py | 105 ++-----
airflow/providers/apache/hdfs/provider.yaml | 15 +-
airflow/providers/apache/hdfs/sensors/hdfs.py | 185 ++----------
airflow/providers/cncf/kubernetes/CHANGELOG.rst | 8 +
.../providers/cncf/kubernetes/hooks/kubernetes.py | 19 +-
airflow/providers/cncf/kubernetes/operators/pod.py | 22 +-
airflow/providers/common/sql/operators/sql.py | 7 +-
airflow/providers/common/sql/provider.yaml | 1 +
.../databricks/operators/databricks_sql.py | 3 +
airflow/providers/databricks/provider.yaml | 3 +-
.../databricks/sensors/databricks_partition.py | 228 +++++++++++++++
airflow/providers/elasticsearch/provider.yaml | 7 +-
airflow/providers/google/cloud/hooks/bigquery.py | 12 +-
.../cloud/hooks/vertex_ai/batch_prediction_job.py | 12 +
.../providers/google/cloud/operators/bigquery.py | 62 +++-
.../operators/vertex_ai/batch_prediction_job.py | 14 +
.../providers/google/cloud/transfers/s3_to_gcs.py | 123 ++++----
.../providers/google/cloud/triggers/bigquery.py | 13 +-
airflow/providers/google/provider.yaml | 2 +-
.../microsoft/azure/operators/data_factory.py | 38 ++-
airflow/providers/mongo/hooks/mongo.py | 37 ++-
airflow/providers/mongo/provider.yaml | 4 +-
.../providers/openlineage/extractors/__init__.py | 4 +
airflow/providers/openlineage/extractors/bash.py | 72 +++++
.../providers/openlineage/extractors/manager.py | 65 ++++-
airflow/providers/openlineage/extractors/python.py | 85 ++++++
airflow/providers/openlineage/plugins/adapter.py | 2 +-
airflow/providers/openlineage/utils/utils.py | 18 +-
.../oracle/example_dags}/__init__.py | 0
.../oracle/example_dags/example_oracle.py | 61 ++++
.../providers/pagerduty/hooks/pagerduty_events.py | 3 +-
.../notifications/{slack_notifier.py => slack.py} | 2 +-
.../slack/notifications/slack_notifier.py | 80 +-----
airflow/sensors/__init__.py | 3 -
airflow/sensors/bash.py | 2 +-
airflow/settings.py | 14 +-
airflow/utils/log/file_task_handler.py | 17 +-
airflow/utils/log/secrets_masker.py | 4 +-
airflow/www/extensions/init_manifest_files.py | 2 +-
airflow/www/forms.py | 3 +-
airflow/www/static/js/api/useTaskLog.ts | 1 -
airflow/www/static/js/dag/details/FilterTasks.tsx | 1 +
.../www/static/js/dag/details/dagRun/ClearRun.tsx | 1 +
.../www/static/js/dag/details/dagRun/MarkRunAs.tsx | 1 +
airflow/www/static/js/dag/details/index.tsx | 11 +-
.../js/dag/details/taskInstance/Logs/index.tsx | 19 +-
.../taskInstance/taskActions/ClearInstance.tsx | 7 +-
airflow/www/templates/airflow/dag_audit_log.html | 1 -
airflow/www/validators.py | 21 ++
dev/README_RELEASE_AIRFLOW.md | 15 +
dev/README_RELEASE_HELM_CHART.md | 15 +
dev/README_RELEASE_PROVIDER_PACKAGES.md | 15 +
.../airflow_breeze/commands/developer_commands.py | 72 ++++-
.../commands/developer_commands_config.py | 14 +-
.../commands/release_management_commands.py | 26 +-
.../airflow_breeze/commands/testing_commands.py | 19 ++
.../commands/testing_commands_config.py | 2 +
dev/breeze/src/airflow_breeze/global_constants.py | 6 +-
dev/breeze/src/airflow_breeze/utils/coertions.py | 6 +
.../airflow_breeze/utils/docker_command_utils.py | 2 +-
dev/breeze/src/airflow_breeze/utils/parallel.py | 2 +-
dev/breeze/src/airflow_breeze/utils/path_utils.py | 1 +
.../airflow_breeze/utils/provider_dependencies.py | 52 ++++
dev/breeze/src/airflow_breeze/utils/run_tests.py | 12 +-
.../src/airflow_breeze/utils/selective_checks.py | 22 +-
dev/breeze/tests/test_provider_dependencies.py | 47 +++
docker_tests/command_utils.py | 15 +-
docker_tests/docker_tests_utils.py | 9 +-
docker_tests/test_docker_compose_quick_start.py | 59 +++-
.../operators/emr/emr.rst | 8 +
.../connections.rst | 9 +-
.../operators/hdfs.rst | 74 -----
.../operators/index.rst | 1 -
.../operators/sql.rst | 49 +++-
docs/apache-airflow-providers-google/index.rst | 2 +-
.../operators/cloud/bigquery.rst | 7 +-
docs/apache-airflow-providers-mongo/index.rst | 2 +-
docs/apache-airflow-providers-oracle/index.rst | 2 +
.../operators/index.rst | 78 +++++
.../notifications/slack_notifier_howto_guide.rst | 4 +-
.../logging-monitoring/callbacks.rst | 9 +
docs/apache-airflow/core-concepts/dags.rst | 2 +-
docs/apache-airflow/img/airflow_erd.sha256 | 2 +-
docs/apache-airflow/img/airflow_erd.svg | 136 ++++-----
docs/apache-airflow/migrations-ref.rst | 4 +-
generated/provider_dependencies.json | 11 +-
images/breeze/output-commands-hash.txt | 6 +-
images/breeze/output_static-checks.svg | 158 ++++++----
.../breeze/output_testing_docker-compose-tests.svg | 54 +++-
newsfragments/30762.improvement.rst | 1 +
newsfragments/31277.significant.rst | 8 +
pyproject.toml | 2 +
.../in_container/run_provider_yaml_files_check.py | 16 +-
scripts/in_container/verify_providers.py | 1 +
setup.cfg | 2 +-
setup.py | 11 +-
tests/always/test_project_structure.py | 17 ++
tests/executors/test_kubernetes_executor.py | 7 +-
tests/jobs/test_base_job.py | 23 ++
tests/models/test_trigger.py | 31 ++
tests/providers/amazon/aws/hooks/test_s3.py | 4 +-
.../amazon/aws/log/test_s3_task_handler.py | 1 +
.../amazon/aws/operators/test_emr_serverless.py | 26 ++
.../aws/operators/test_sagemaker_transform.py | 17 +-
tests/providers/amazon/aws/sensors/test_ec2.py | 19 ++
.../amazon/aws/transfers/test_mongo_to_s3.py | 12 +-
tests/providers/amazon/aws/triggers/test_ec2.py | 103 +++++++
tests/providers/apache/hdfs/hooks/test_hdfs.py | 86 ------
tests/providers/apache/hdfs/sensors/test_hdfs.py | 320 ---------------------
tests/providers/apache/hive/sensors/test_hdfs.py | 38 ---
.../{test_kubernetes_pod.py => test_kubernetes.py} | 33 +++
.../cncf/kubernetes/operators/test_pod.py | 36 +++
.../databricks/operators/test_databricks_sql.py | 23 +-
...abricks_sql.py => test_databricks_partition.py} | 60 ++--
.../providers/google/cloud/hooks/test_bigquery.py | 26 ++
.../google/cloud/operators/test_bigquery.py | 37 +--
.../google/cloud/operators/test_vertex_ai.py | 7 +
.../google/cloud/transfers/test_s3_to_gcs.py | 146 +++++++++-
tests/providers/google/cloud/triggers/test_gcs.py | 6 +-
.../microsoft/azure/log/test_wasb_task_handler.py | 2 +-
.../azure/operators/test_azure_data_factory.py | 51 +++-
tests/providers/mongo/hooks/test_mongo.py | 12 +-
tests/providers/openlineage/__init__.py | 3 -
.../providers/openlineage/extractors}/__init__.py | 0
.../openlineage/extractors/test_bash_extractor.py | 113 ++++++++
.../extractors/test_python_extractor.py | 148 ++++++++++
tests/providers/openlineage/log_config.py | 3 -
.../providers/openlineage/plugins/test_listener.py | 3 -
.../plugins/test_openlineage_adapter.py | 1 -
tests/providers/openlineage/plugins/test_utils.py | 3 -
.../{test_slack_notifier.py => test_slack.py} | 39 +--
tests/providers/slack/operators/test_slack.py | 17 --
.../providers/amazon/aws/example_dynamodb_to_s3.py | 4 +-
.../aws/example_eks_with_fargate_in_one_step.py | 17 +-
.../amazon/aws/example_eks_with_fargate_profile.py | 17 +-
.../aws/example_eks_with_nodegroup_in_one_step.py | 17 +-
.../amazon/aws/example_eks_with_nodegroups.py | 21 +-
.../system/providers/amazon/aws/utils/k8s.py | 42 ++-
.../databricks/example_databricks_sensors.py | 26 +-
.../google/cloud/gcs/example_s3_to_gcs.py | 6 +-
tests/test_utils/hdfs_utils.py | 168 -----------
tests/utils/test_helpers.py | 2 +-
tests/utils/test_log_handlers.py | 59 +++-
tests/www/test_validators.py | 35 +++
tests/www/views/test_views_connection.py | 11 +
181 files changed, 3258 insertions(+), 1815 deletions(-)
copy airflow/migrations/versions/{0034_1_10_0_index_taskfail.py =>
0125_2_7_0_add_index_to_task_instance_table.py} (65%)
copy airflow/providers/amazon/aws/{sensors => triggers}/ec2.py (50%)
create mode 100644 airflow/providers/databricks/sensors/databricks_partition.py
create mode 100644 airflow/providers/openlineage/extractors/bash.py
create mode 100644 airflow/providers/openlineage/extractors/python.py
copy airflow/{api_connexion => providers/oracle/example_dags}/__init__.py
(100%)
create mode 100644 airflow/providers/oracle/example_dags/example_oracle.py
copy airflow/providers/slack/notifications/{slack_notifier.py => slack.py}
(98%)
create mode 100644 dev/breeze/src/airflow_breeze/utils/provider_dependencies.py
create mode 100644 dev/breeze/tests/test_provider_dependencies.py
delete mode 100644 docs/apache-airflow-providers-apache-hdfs/operators/hdfs.rst
create mode 100644 docs/apache-airflow-providers-oracle/operators/index.rst
create mode 100644 newsfragments/30762.improvement.rst
create mode 100644 newsfragments/31277.significant.rst
create mode 100644 tests/providers/amazon/aws/triggers/test_ec2.py
delete mode 100644 tests/providers/apache/hdfs/hooks/test_hdfs.py
delete mode 100644 tests/providers/apache/hdfs/sensors/test_hdfs.py
delete mode 100644 tests/providers/apache/hive/sensors/test_hdfs.py
rename tests/providers/cncf/kubernetes/hooks/{test_kubernetes_pod.py =>
test_kubernetes.py} (95%)
copy tests/providers/databricks/sensors/{test_databricks_sql.py =>
test_databricks_partition.py} (60%)
copy {airflow/api_connexion =>
tests/providers/openlineage/extractors}/__init__.py (100%)
create mode 100644
tests/providers/openlineage/extractors/test_bash_extractor.py
create mode 100644
tests/providers/openlineage/extractors/test_python_extractor.py
rename tests/providers/slack/notifications/{test_slack_notifier.py =>
test_slack.py} (71%)
copy airflow/migrations/versions/0025_1_8_2_add_ti_job_id_index.py =>
tests/system/providers/amazon/aws/utils/k8s.py (55%)
