Re: [PR] Add method to get metadata from GCS blob in GCSHook [airflow]

[via GitHub]

jalengg commented on code in PR #38398:
URL: https://github.com/apache/airflow/pull/38398#discussion_r1536565254


##
airflow/providers/google/cloud/hooks/gcs.py:
##
@@ -1006,6 +1006,27 @@ def get_md5hash(self, bucket_name: str, object_name: 
str) -> str:
 self.log.info("The md5Hash of %s is %s", object_name, blob_md5hash)
 return blob_md5hash
 
+def get_metadata(self, bucket_name: str, object_name: str) -> dict | None:
+"""
+Get the metadata of an object in Google Cloud Storage.
+
+:param bucket_name: Name of the Google Cloud Storage bucket where the 
object is.
+:param object_name: The name of the object containing the desired 
metadata
+:return: The metadata associated with the object
+"""
+self.log.info("Retrieving the metadata dict of object (%s) in bucket 
(%s)", object_name, bucket_name)
+client = self.get_conn()
+bucket = client.bucket(bucket_name)
+blob = bucket.get_blob(blob_name=object_name)
+if blob is None:
+raise ValueError("Object (%s) not found in bucket (%s)", 
object_name, bucket_name)

Review Comment:
   My intention was to surface the error if the blob doesn't exist because it 
prevents a nonetype attribute error. It may be  reasonable that the blob exists 
and the user will want to get metadata even if it's empty (for this return 
None), but in my opinion not reasonable for the user to supply an empty or 
non-existent blob to the function. So, I think it's appropriate to raise an 
exception and prevent the nonetype attribute error. The existing 
`get_blob_update_time` method in GCSHook handles the empty blob in the same way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] SparkKubernetesOperator not retrieves logs from the driver pod and displays them in the Airflow UI. [airflow]

[via GitHub]

sudohainguyen commented on issue #37681:
URL: https://github.com/apache/airflow/issues/37681#issuecomment-2016329205

   hey can we expect this will be fixed in the next release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add method to get metadata from GCS blob in GCSHook [airflow]

[via GitHub]

dirrao commented on code in PR #38398:
URL: https://github.com/apache/airflow/pull/38398#discussion_r1536551567


##
airflow/providers/google/cloud/hooks/gcs.py:
##
@@ -1006,6 +1006,27 @@ def get_md5hash(self, bucket_name: str, object_name: 
str) -> str:
 self.log.info("The md5Hash of %s is %s", object_name, blob_md5hash)
 return blob_md5hash
 
+def get_metadata(self, bucket_name: str, object_name: str) -> dict | None:
+"""
+Get the metadata of an object in Google Cloud Storage.
+
+:param bucket_name: Name of the Google Cloud Storage bucket where the 
object is.
+:param object_name: The name of the object containing the desired 
metadata
+:return: The metadata associated with the object
+"""
+self.log.info("Retrieving the metadata dict of object (%s) in bucket 
(%s)", object_name, bucket_name)
+client = self.get_conn()
+bucket = client.bucket(bucket_name)
+blob = bucket.get_blob(blob_name=object_name)
+if blob is None:
+raise ValueError("Object (%s) not found in bucket (%s)", 
object_name, bucket_name)

Review Comment:
   why are we throwing error instead of return None here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: BigQueryCheckOperator skip value and error check in deferrable mode [airflow]

[via GitHub]

dirrao commented on code in PR #38408:
URL: https://github.com/apache/airflow/pull/38408#discussion_r1536548308


##
airflow/providers/google/cloud/operators/bigquery.py:
##
@@ -322,8 +322,26 @@ def execute(self, context: Context):
 ),
 method_name="execute_complete",
 )
+self._handle_job_error(job)
+# job.result() returns a RowIterator. Mypy expects an instance of 
SupportsNext[Any] for
+# the next() call which the RowIterator does not resemble to. 
Hence, ignore the arg-type error.
+records = next(job.result())  # type: ignore[arg-type]

Review Comment:
   next call call might returns a row object not row iterator. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch modify-docker-tests updated (ff0fc7568a -> a36fb6eb82)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


from ff0fc7568a Respect docker-tests requirements (use venv)
 add a36fb6eb82 Create temporary virtualenv as context manager

No new revisions were added by this update.

Summary of changes:
 dev/breeze/src/airflow_breeze/utils/run_tests.py   | 18 +++-
 .../src/airflow_breeze/utils/virtualenv_utils.py   | 49 +++---
 2 files changed, 49 insertions(+), 18 deletions(-)

(airflow) branch modify-docker-tests updated (fdbb7e3d37 -> ff0fc7568a)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


from fdbb7e3d37 Use `python-on-whales` in docker tests
 add dec65d5400 Use image fixture for determine default version
 add ff0fc7568a Respect docker-tests requirements (use venv)

No new revisions were added by this update.

Summary of changes:
 .../commands/release_management_commands.py| 35 ++---
 dev/breeze/src/airflow_breeze/utils/run_tests.py   | 43 ++-
 .../src/airflow_breeze/utils/virtualenv_utils.py   | 61 ++
 .../providers/google => docker_tests}/conftest.py  |  9 ++--
 docker_tests/docker_utils.py   |  6 +--
 docker_tests/test_ci_image.py  |  4 +-
 docker_tests/test_docker_compose_quick_start.py|  6 +--
 docker_tests/test_prod_image.py| 46 +---
 8 files changed, 134 insertions(+), 76 deletions(-)
 create mode 100644 dev/breeze/src/airflow_breeze/utils/virtualenv_utils.py
 copy {tests/system/providers/google => docker_tests}/conftest.py (83%)

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

lyuyunbo commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016229746

   > ```
   > change()
   > try:
   >do stuff
   > finally:
   >restore()
   > ```
   
   Ha! I misunderstood which try-catch block you were mentioning before. 
Updated, this is indeed a good catch, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

[via GitHub]

kaxil commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2016216787

   >Can everyone in this thread make an experiment, disable it and report back 
here - at least this will give some clue on where the problems might happen.
   
   Yes please, @alex-magno Could you try with disabling mini-scheduler?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] add "return" statement to "yield" within a while loop in core triggers [airflow]

[via GitHub]

Lee-W commented on PR #38389:
URL: https://github.com/apache/airflow/pull/38389#issuecomment-2016215062

   > related to that #38353 ?
   
   I think it's somewhat related but not 100% sure this is the root cause


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Support configuring multiple namespace for multiNamespaceMode [airflow]

[via GitHub]

github-actions[bot] commented on PR #35639:
URL: https://github.com/apache/airflow/pull/35639#issuecomment-2016205670

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add type hints for AWS provider [airflow]

[via GitHub]

github-actions[bot] commented on PR #35966:
URL: https://github.com/apache/airflow/pull/35966#issuecomment-2016205635

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [WIP] Mapping tasks from ObjectStoragePath [airflow]

[via GitHub]

github-actions[bot] commented on PR #36319:
URL: https://github.com/apache/airflow/pull/36319#issuecomment-2016205587

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Use `python-on-whales` in docker tests [airflow]

[via GitHub]

Taragolis opened a new pull request, #38421:
URL: https://github.com/apache/airflow/pull/38421

   
   
   
   
   This attempt to reduce number of subprocess usage and switch to 
[`python-on-whales`](https://github.com/gabrieldemarmiesse/python-on-whales) 
where it possible.
   
   The main benefit that we do not need to parse output manually instead of we 
could work directly with objects (Spoiler Alert: Under the hood 
`python-on-whales` do that). In addition debugging become more easier, because 
you might inspect objects retrieved from `python-on-whales`.
   
   There is no any speedup on tests expected, maybe even increase a bit but I 
guess it shouldn't be noticeable.
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) 01/01: Use `python-on-whales` in docker tests

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a commit to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit fdbb7e3d37fad146a4dc16f007dde58d0fbeae69
Author: Andrey Anshin 
AuthorDate: Sat Mar 23 03:40:48 2024 +0400

Use `python-on-whales` in docker tests
---
 docker_tests/constants.py  |   5 +
 docker_tests/docker_tests_utils.py | 105 ---
 docker_tests/docker_utils.py   | 141 +
 docker_tests/requirements.txt  |   4 +-
 docker_tests/test_ci_image.py  |  28 ++--
 docker_tests/test_docker_compose_quick_start.py| 122 ++
 .../test_examples_of_prod_image_building.py|  19 ++-
 docker_tests/test_prod_image.py|  88 -
 8 files changed, 237 insertions(+), 275 deletions(-)

diff --git a/docker_tests/constants.py b/docker_tests/constants.py
index db79d1d862..83c77e2fb4 100644
--- a/docker_tests/constants.py
+++ b/docker_tests/constants.py
@@ -16,6 +16,11 @@
 # under the License.
 from __future__ import annotations
 
+import os
 from pathlib import Path
 
 SOURCE_ROOT = Path(__file__).resolve().parents[1]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
+DEFAULT_DOCKER_IMAGE = 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
diff --git a/docker_tests/docker_tests_utils.py 
b/docker_tests/docker_tests_utils.py
deleted file mode 100644
index 7eea98e9bd..00
--- a/docker_tests/docker_tests_utils.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-from __future__ import annotations
-
-import os
-
-from docker_tests.command_utils import run_command
-
-DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
-
-docker_image = os.environ.get(
-"DOCKER_IMAGE", 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
-)
-
-print("Using docker image: ", docker_image)
-
-
-def run_bash_in_docker(bash_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"--entrypoint",
-"/bin/bash",
-docker_image,
-"-c",
-bash_script,
-]
-return run_command(docker_command, **kwargs)
-
-
-def run_python_in_docker(python_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"-e",
-"PYTHONDONTWRITEBYTECODE=true",
-docker_image,
-"python",
-"-c",
-python_script,
-]
-return run_command(docker_command, **kwargs)
-
-
-def display_dependency_conflict_message():
-print(
-"""
-* Beginning of the instructions 
-
-The image did not pass 'pip check' verification. This means that there are 
some conflicting dependencies
-in the image.
-
-It can mean one of those:
-
-1) The main is currently broken (other PRs will fail with the same error)
-2) You changed some dependencies in pyproject.toml (either manually or 
automatically by pre-commit)
-   and they are conflicting.
-
-
-
-In case 1) - apologies for the trouble.Please let committers know and they 
will fix it. You might
-be asked to rebase to the latest main after the problem is fixed.
-
-In case 2) - Follow the steps below:
-
-* try to build CI and then PROD image locally with breeze, adding 
--upgrade-to-newer-dependencies flag
-  (repeat it for all python versions)
-
-CI image:
-
- breeze ci-image build --upgrade-to-newer-dependencies --python 3.8
-
-Production image:
-
- breeze ci-image build --production-image --upgrade-to-newer-dependencies 
--python 3.8
-
-* You will see error messages there telling which requirements are conflicting 
and which packages caused the
-  conflict. Add the limitation that caused the conflict to 
EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS
-  variable in Dockerfile.ci. Note that the limitations might be different for 
Dockerfile.ci and Dockerfile
-  because not all packages are installed by default in the PROD 

(airflow) branch modify-docker-tests updated (393af8e7bf -> fdbb7e3d37)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 393af8e7bf Use `python-on-whales` in docker tests
 new fdbb7e3d37 Use `python-on-whales` in docker tests

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (393af8e7bf)
\
 N -- N -- N   refs/heads/modify-docker-tests (fdbb7e3d37)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docker_tests/docker_utils.py | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

(airflow) 01/01: Use `python-on-whales` in docker tests

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a commit to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit d9549bbadeee2193037e4a2ae88bf255a315bb09
Author: Andrey Anshin 
AuthorDate: Sat Mar 23 03:40:48 2024 +0400

Use `python-on-whales` in docker tests
---
 docker_tests/constants.py  |   5 +
 .../{docker_tests_utils.py => docker_utils.py} | 102 ++---
 docker_tests/requirements.txt  |   4 +-
 docker_tests/test_ci_image.py  |  28 ++---
 docker_tests/test_docker_compose_quick_start.py| 122 +++--
 .../test_examples_of_prod_image_building.py|  19 ++--
 docker_tests/test_prod_image.py|  88 +--
 7 files changed, 160 insertions(+), 208 deletions(-)

diff --git a/docker_tests/constants.py b/docker_tests/constants.py
index db79d1d862..83c77e2fb4 100644
--- a/docker_tests/constants.py
+++ b/docker_tests/constants.py
@@ -16,6 +16,11 @@
 # under the License.
 from __future__ import annotations
 
+import os
 from pathlib import Path
 
 SOURCE_ROOT = Path(__file__).resolve().parents[1]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
+DEFAULT_DOCKER_IMAGE = 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
diff --git a/docker_tests/docker_tests_utils.py b/docker_tests/docker_utils.py
similarity index 51%
rename from docker_tests/docker_tests_utils.py
rename to docker_tests/docker_utils.py
index 7eea98e9bd..2eb67c9c88 100644
--- a/docker_tests/docker_tests_utils.py
+++ b/docker_tests/docker_utils.py
@@ -16,50 +16,49 @@
 # under the License.
 from __future__ import annotations
 
-import os
-
-from docker_tests.command_utils import run_command
+from time import monotonic, sleep
+
+from python_on_whales import docker
+from python_on_whales.exceptions import NoSuchContainer
+
+from docker_tests.constants import DEFAULT_DOCKER_IMAGE
+
+
+def run_cmd_in_docker(
+cmd: list[str] | None = None,
+docker_image: str = DEFAULT_DOCKER_IMAGE,
+entrypoint: str | None = None,
+envs: dict[str, str] | None = None,
+remove: bool = True,
+**kwargs,
+):
+kwargs.pop("image", None)
+cmd = cmd or []
+envs = envs or {}
+return docker.run(
+image=docker_image,
+entrypoint=entrypoint,
+command=cmd,
+remove=remove,
+envs={"COLUMNS": "180", **envs},
+**kwargs,
+)
 
-DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
 
-docker_image = os.environ.get(
-"DOCKER_IMAGE", 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
-)
+def run_bash_in_docker(bash_script: str, **kwargs):
+kwargs.pop("entrypoint", None)
+return run_cmd_in_docker(cmd=["-c", bash_script], entrypoint="/bin/bash", 
**kwargs)
 
-print("Using docker image: ", docker_image)
 
+def run_python_in_docker(python_script, **kwargs):
+kwargs.pop("entrypoint", None)
+envs = {"PYTHONDONTWRITEBYTECODE": "true", **kwargs.pop("envs", {})}
+return run_cmd_in_docker(cmd=["python", "-c", python_script], envs=envs, 
**kwargs)
 
-def run_bash_in_docker(bash_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"--entrypoint",
-"/bin/bash",
-docker_image,
-"-c",
-bash_script,
-]
-return run_command(docker_command, **kwargs)
 
-
-def run_python_in_docker(python_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"-e",
-"PYTHONDONTWRITEBYTECODE=true",
-docker_image,
-"python",
-"-c",
-python_script,
-]
-return run_command(docker_command, **kwargs)
+def run_airflow_cmd_in_docker(cmd: list[str] | None = None, **kwargs):
+kwargs.pop("entrypoint", None)
+return run_cmd_in_docker(cmd=["airflow", *(cmd or [])], **kwargs)
 
 
 def display_dependency_conflict_message():
@@ -103,3 +102,30 @@ Production image:
 * End of the instructions 
 """
 )
+
+
+def wait_for_container(container_id: str, timeout: int = 300):
+print(f"Waiting for container: [{container_id}] for {timeout} more 
seconds.")
+start_time = monotonic()
+while True:
+if timeout != 0 and monotonic() - start_time > timeout:
+err_msg = f"Timeout. The operation takes longer than the maximum 
waiting time ({timeout}s)"
+raise TimeoutError(err_msg)
+
+try:
+container = docker.container.inspect("container_id")
+except NoSuchContainer:
+sleep(5)
+continue
+container.export()
+if (state := container.state).status in ("running", "restarting"):
+if state.health_status is None or 

(airflow) branch modify-docker-tests created (now d9549bbade)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at d9549bbade Use `python-on-whales` in docker tests

This branch includes the following new commits:

 new d9549bbade Use `python-on-whales` in docker tests

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) 01/01: Use `python-on-whales` in docker tests

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a commit to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 393af8e7bfcfe72d4500bb25153fe6228bf7d017
Author: Andrey Anshin 
AuthorDate: Sat Mar 23 03:40:48 2024 +0400

Use `python-on-whales` in docker tests
---
 docker_tests/constants.py  |   5 +
 .../{docker_tests_utils.py => docker_utils.py} | 102 ++---
 docker_tests/requirements.txt  |   4 +-
 docker_tests/test_ci_image.py  |  28 ++---
 docker_tests/test_docker_compose_quick_start.py| 122 +++--
 .../test_examples_of_prod_image_building.py|  19 ++--
 docker_tests/test_prod_image.py|  88 +--
 7 files changed, 160 insertions(+), 208 deletions(-)

diff --git a/docker_tests/constants.py b/docker_tests/constants.py
index db79d1d862..83c77e2fb4 100644
--- a/docker_tests/constants.py
+++ b/docker_tests/constants.py
@@ -16,6 +16,11 @@
 # under the License.
 from __future__ import annotations
 
+import os
 from pathlib import Path
 
 SOURCE_ROOT = Path(__file__).resolve().parents[1]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
+DEFAULT_DOCKER_IMAGE = 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
diff --git a/docker_tests/docker_tests_utils.py b/docker_tests/docker_utils.py
similarity index 51%
rename from docker_tests/docker_tests_utils.py
rename to docker_tests/docker_utils.py
index 7eea98e9bd..2eb67c9c88 100644
--- a/docker_tests/docker_tests_utils.py
+++ b/docker_tests/docker_utils.py
@@ -16,50 +16,49 @@
 # under the License.
 from __future__ import annotations
 
-import os
-
-from docker_tests.command_utils import run_command
+from time import monotonic, sleep
+
+from python_on_whales import docker
+from python_on_whales.exceptions import NoSuchContainer
+
+from docker_tests.constants import DEFAULT_DOCKER_IMAGE
+
+
+def run_cmd_in_docker(
+cmd: list[str] | None = None,
+docker_image: str = DEFAULT_DOCKER_IMAGE,
+entrypoint: str | None = None,
+envs: dict[str, str] | None = None,
+remove: bool = True,
+**kwargs,
+):
+kwargs.pop("image", None)
+cmd = cmd or []
+envs = envs or {}
+return docker.run(
+image=docker_image,
+entrypoint=entrypoint,
+command=cmd,
+remove=remove,
+envs={"COLUMNS": "180", **envs},
+**kwargs,
+)
 
-DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.8"
 
-docker_image = os.environ.get(
-"DOCKER_IMAGE", 
f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
-)
+def run_bash_in_docker(bash_script: str, **kwargs):
+kwargs.pop("entrypoint", None)
+return run_cmd_in_docker(cmd=["-c", bash_script], entrypoint="/bin/bash", 
**kwargs)
 
-print("Using docker image: ", docker_image)
 
+def run_python_in_docker(python_script, **kwargs):
+kwargs.pop("entrypoint", None)
+envs = {"PYTHONDONTWRITEBYTECODE": "true", **kwargs.pop("envs", {})}
+return run_cmd_in_docker(cmd=["python", "-c", python_script], envs=envs, 
**kwargs)
 
-def run_bash_in_docker(bash_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"--entrypoint",
-"/bin/bash",
-docker_image,
-"-c",
-bash_script,
-]
-return run_command(docker_command, **kwargs)
 
-
-def run_python_in_docker(python_script, **kwargs):
-docker_command = [
-"docker",
-"run",
-"--rm",
-"-e",
-"COLUMNS=180",
-"-e",
-"PYTHONDONTWRITEBYTECODE=true",
-docker_image,
-"python",
-"-c",
-python_script,
-]
-return run_command(docker_command, **kwargs)
+def run_airflow_cmd_in_docker(cmd: list[str] | None = None, **kwargs):
+kwargs.pop("entrypoint", None)
+return run_cmd_in_docker(cmd=["airflow", *(cmd or [])], **kwargs)
 
 
 def display_dependency_conflict_message():
@@ -103,3 +102,30 @@ Production image:
 * End of the instructions 
 """
 )
+
+
+def wait_for_container(container_id: str, timeout: int = 300):
+print(f"Waiting for container: [{container_id}] for {timeout} more 
seconds.")
+start_time = monotonic()
+while True:
+if timeout != 0 and monotonic() - start_time > timeout:
+err_msg = f"Timeout. The operation takes longer than the maximum 
waiting time ({timeout}s)"
+raise TimeoutError(err_msg)
+
+try:
+container = docker.container.inspect("container_id")
+except NoSuchContainer:
+sleep(5)
+continue
+container.export()
+if (state := container.state).status in ("running", "restarting"):
+if state.health_status is None or 

(airflow) branch modify-docker-tests updated (d9549bbade -> 393af8e7bf)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch modify-docker-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard d9549bbade Use `python-on-whales` in docker tests
 new 393af8e7bf Use `python-on-whales` in docker tests

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (d9549bbade)
\
 N -- N -- N   refs/heads/modify-docker-tests (393af8e7bf)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

potiuk commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016154715

   
   ```
   change()
   try:
  do stuff
   finally:
  restore()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Mysql change xcom value col type for MySQL backend [airflow]

[via GitHub]

kaxil commented on PR #38401:
URL: https://github.com/apache/airflow/pull/38401#issuecomment-2016148389

   >If it does, and it results in a slow migration, I really wonder if we 
should not do this and instruct users to set up an object storage xcom backend 
instead. Thoughts?
   
   The current size is low per row - 64kb for MySQL, so increasing to 64 MB 
should be supported without Custom XCom backend


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch constraints-main updated: Updating constraints. Github run id:8397236584

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 3aeee405a2 Updating constraints. Github run id:8397236584
3aeee405a2 is described below

commit 3aeee405a269662970a168830b926395281bd050
Author: Automated GitHub Actions commit 
AuthorDate: Fri Mar 22 23:22:46 2024 +

Updating constraints. Github run id:8397236584

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
'refs/heads/main' in the 'apache/airflow' repository with commit sha 
88ea87d3d016beb980b73b3ec9ee1a0f5360170f.

The action that build those constraints can be found at 
https://github.com/apache/airflow/actions/runs/8397236584/

The image tag used for that build was: 
88ea87d3d016beb980b73b3ec9ee1a0f5360170f. You can enter Breeze environment
with this image by running 'breeze shell --image-tag 
88ea87d3d016beb980b73b3ec9ee1a0f5360170f'

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 18 +-
 constraints-3.11.txt  | 18 +-
 constraints-3.12.txt  | 18 +-
 constraints-3.8.txt   | 18 +-
 constraints-3.9.txt   | 18 +-
 constraints-no-providers-3.10.txt |  8 
 constraints-no-providers-3.11.txt |  8 
 constraints-no-providers-3.12.txt |  8 
 constraints-no-providers-3.8.txt  |  8 
 constraints-no-providers-3.9.txt  |  8 
 constraints-source-providers-3.10.txt | 18 +-
 constraints-source-providers-3.11.txt | 18 +-
 constraints-source-providers-3.12.txt | 18 +-
 constraints-source-providers-3.8.txt  | 18 +-
 constraints-source-providers-3.9.txt  | 18 +-
 15 files changed, 110 insertions(+), 110 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 6c7dabe89f..99a66eb67a 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,6 +1,6 @@
 
 #
-# This constraints file was automatically generated on 
2024-03-22T12:01:38.966348
+# This constraints file was automatically generated on 
2024-03-22T23:03:01.290339
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -186,7 +186,7 @@ apispec==6.6.0
 apprise==1.7.4
 argcomplete==3.2.3
 asana==3.2.2
-asgiref==3.8.0
+asgiref==3.8.1
 asn1crypto==1.5.1
 astroid==2.15.8
 asttokens==2.4.1
@@ -321,14 +321,14 @@ google-cloud-batch==0.17.14
 google-cloud-bigquery-datatransfer==3.15.1
 google-cloud-bigquery==3.19.0
 google-cloud-bigtable==2.23.0
-google-cloud-build==3.23.3
+google-cloud-build==3.24.0
 google-cloud-compute==1.18.0
-google-cloud-container==2.43.0
+google-cloud-container==2.44.0
 google-cloud-core==2.4.1
-google-cloud-datacatalog==3.18.3
+google-cloud-datacatalog==3.19.0
 google-cloud-dataflow-client==0.8.10
 google-cloud-dataform==0.5.9
-google-cloud-dataplex==1.12.3
+google-cloud-dataplex==1.13.0
 google-cloud-dataproc-metastore==1.15.3
 google-cloud-dataproc==5.9.3
 google-cloud-dlp==3.16.0
@@ -343,7 +343,7 @@ google-cloud-pubsub==2.20.3
 google-cloud-redis==2.15.3
 google-cloud-resource-manager==1.12.3
 google-cloud-run==0.10.5
-google-cloud-secret-manager==2.18.3
+google-cloud-secret-manager==2.19.0
 google-cloud-spanner==3.44.0
 google-cloud-speech==2.25.1
 google-cloud-storage-transfer==1.11.3
@@ -506,7 +506,7 @@ pendulum==3.0.0
 pexpect==4.9.0
 pgvector==0.2.5
 pinecone-client==2.2.4
-pinotdb==5.1.4
+pinotdb==5.1.5
 pipdeptree==2.16.1
 pipx==1.4.3
 pkginfo==1.10.0
@@ -705,7 +705,7 @@ universal_pathlib==0.2.2
 uritemplate==4.1.1
 urllib3==2.0.7
 userpath==1.9.2
-uv==0.1.22
+uv==0.1.24
 validators==0.23.2
 vertica-python==1.3.8
 vine==5.1.0
diff --git a/constraints-3.11.txt b/constraints-3.11.txt
index 0536f2a63e..331d2c1726 100644
--- a/constraints-3.11.txt
+++ b/constraints-3.11.txt
@@ -1,6 +1,6 @@
 
 #
-# This constraints file was automatically generated on 
2024-03-22T12:01:38.658655
+# This constraints file was automatically generated on 
2024-03-22T23:03:01.224815
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -186,7 +186,7 @@ apispec==6.6.0
 

Re: [PR] [AIP-49] Airflow OpenTelemetry Provider [airflow]

[via GitHub]

hussein-awala commented on PR #37989:
URL: https://github.com/apache/airflow/pull/37989#issuecomment-2016109464

   The PR was broken with around 100 unrelated commits, I just squashed all 
related commits into a single commit and rebased `main` as it was impossible to 
review the PR. Could you please check it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

lyuyunbo commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016091345

   > Still try/finally not solved
   
   Can you be more specific please @potiuk? Because we indeed added 
`_restore_log_formatter` into the except clause, which will revert the changes 
if formatters are not fully updated to the customized one.
   
   If you are referring to the original suggestion:
   
   > Maybe I am nissing something I thought about try/finally where 
_restore_log_formatter will be called in the finally clause.
   
   I think we've confused something here, because `_restore_log_formatter` will 
roll back all formatters no matter the try block succeeded or not, and makes 
this feature not working anymore.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Migrate to connexion v3 [airflow]

[via GitHub]

Satoshi-Sh commented on PR #37638:
URL: https://github.com/apache/airflow/pull/37638#issuecomment-2016090236

   I've been working on `tests/api_connexion/test_cors.py`. It would be great 
if you could give me some inputs on this.
   
   You can find the details 
[here](https://github.com/sudiptob2/airflow/pull/33) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] feat: Add DbtCloudRetryJobOperator to retry failed dbt job [airflow]

[via GitHub]

andyguwc commented on PR #38001:
URL: https://github.com/apache/airflow/pull/38001#issuecomment-2016065152

   @josh-fell As I think about adding a flag for retry from failure. I can't 
wrap my head around the logic for retry in a deferrable operator. Mind share 
your thoughts on how this could work if the operator is deferrable?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

potiuk commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016055030

   Still try/finally not solved 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[I] Add legend for average task runtime graph [airflow]

[via GitHub]

simond opened a new issue, #38420:
URL: https://github.com/apache/airflow/issues/38420

   ### Description
   
   When clicking on a task in the grid view, a newly-created chart that shows 
the duration of the tasks shows two reference lines. It is not clear what these 
reference lines mean and there should be a legend that describes what they are. 
Is one the average duration and the other the median duration? 
   
   
![image](https://github.com/apache/airflow/assets/421551/9523247f-dd4c-423f-8abd-0a724d0e096c)
   
   I think it would make sense to have a legend either below or beside the 
graph to describe what the reference lines are.
   
   ### Use case/motivation
   
   The motivation is to allow users to understand what the lines on the chart 
represent. 
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add declarative configuration to define custom roles [airflow]

[via GitHub]

potiuk commented on issue #38347:
URL: https://github.com/apache/airflow/issues/38347#issuecomment-2016047870

   > What's currently being done for connections and variables to solve that 
issue? They both already have declarative equivalents.
   
   No - they don't - same problem. When you allow to edit data over the UI and 
want to also upload it at the deployment time, you need to solve the problem of 
data being modified both manually and through declarative sources at deplpyment 
time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add declarative configuration to define custom roles [airflow]

[via GitHub]

potiuk commented on issue #38347:
URL: https://github.com/apache/airflow/issues/38347#issuecomment-2016046849

   > How have they failed?
   
   To solve all the edge cases and the fact that users can modify the DB after 
it has been created and how to consistently mantain source of the truth.t But 
maybe your proposal will fix it. Looking forward to it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] refactor: Refactored __new__ magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators [airflow]

[via GitHub]

potiuk commented on PR #37937:
URL: https://github.com/apache/airflow/pull/37937#issuecomment-2016043868

   @ashb ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated (a7d7ef6433 -> 88ea87d3d0)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from a7d7ef6433 Don't allow defaults other than None in context parameters, 
and improve error message (#38015)
 add 88ea87d3d0 Update UV to latest version released (#38419)

No new revisions were added by this update.

Summary of changes:
 Dockerfile | 2 +-
 Dockerfile.ci  | 4 ++--
 pyproject.toml | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

Re: [PR] Update UV to latest version released [airflow]

[via GitHub]

potiuk merged PR #38419:
URL: https://github.com/apache/airflow/pull/38419


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

lyuyunbo commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2015985430

   > conflicts to resolve when rebasing.
   
   Resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Merge kubernetes pod objects inplace [airflow]

[via GitHub]

potiuk commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2015984516

   > what do you mean by breaking IMHO it's just broken, not breaking - because 
it's impossible to foresee how one pod executing will impact another one, and 
since you have no influence on that, this is indeterministic. Which trades of 
performance with predictability . Or am I wrong?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Improve logging readability with DockerOperator [airflow]

[via GitHub]

potiuk commented on PR #38008:
URL: https://github.com/apache/airflow/pull/38008#issuecomment-2015980402

   conflicts to resolve when rebasing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

[via GitHub]

boring-cyborg[bot] commented on PR #38015:
URL: https://github.com/apache/airflow/pull/38015#issuecomment-2015977471

   Awesome work, congrats on your first merged pull request! You are invited to 
check our [Issue Tracker](https://github.com/apache/airflow/issues) for 
additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Don't allow defaults other than None in context parameters, and improve error message (#38015)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a7d7ef6433 Don't allow defaults other than None in context parameters, 
and improve error message (#38015)
a7d7ef6433 is described below

commit a7d7ef6433a068266b1403e1d94dda2dc85bd634
Author: Kevin Languasco 
AuthorDate: Fri Mar 22 16:44:47 2024 -0500

Don't allow defaults other than None in context parameters, and improve 
error message (#38015)

* feat: don't allow defaults other than None for context parameters

* feat: improve error message when replacing context parameters with None 
breaks signature

* feat: improve error message for unsupported position of context key 
parameter

* style: make stack trace more readable by putting the error message in a 
variable

* feat: add details to exceptions and related test

* fix: put task decorated function outside pytest.raises call
---
 airflow/decorators/base.py  | 29 -
 tests/decorators/test_python.py | 11 +++
 2 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/airflow/decorators/base.py b/airflow/decorators/base.py
index 0b0d921f8c..7adaf8a447 100644
--- a/airflow/decorators/base.py
+++ b/airflow/decorators/base.py
@@ -208,11 +208,38 @@ class DecoratedOperator(BaseOperator):
 # since values for those will be provided when the task is run. Since
 # we're not actually running the function, None is good enough here.
 signature = inspect.signature(python_callable)
+
+# Don't allow context argument defaults other than None to avoid 
ambiguities.
+faulty_parameters = [
+param.name
+for param in signature.parameters.values()
+if param.name in KNOWN_CONTEXT_KEYS and param.default not in 
(None, inspect.Parameter.empty)
+]
+if faulty_parameters:
+message = f"Context key parameter {faulty_parameters[0]} can't 
have a default other than None"
+raise ValueError(message)
+
 parameters = [
 param.replace(default=None) if param.name in KNOWN_CONTEXT_KEYS 
else param
 for param in signature.parameters.values()
 ]
-signature = signature.replace(parameters=parameters)
+try:
+signature = signature.replace(parameters=parameters)
+except ValueError as err:
+message = textwrap.dedent(
+f"""
+The function signature broke while assigning defaults to 
context key parameters.
+
+The decorator is replacing the signature
+> {python_callable.__name__}({', '.join(str(param) for param 
in signature.parameters.values())})
+
+with
+> {python_callable.__name__}({', '.join(str(param) for param 
in parameters)})
+
+which isn't valid: {err}
+"""
+)
+raise ValueError(message) from err
 
 # Check that arguments can be binded. There's a slight difference when
 # we do validation for task-mapping: Since there's no guarantee we can
diff --git a/tests/decorators/test_python.py b/tests/decorators/test_python.py
index fde79a4784..5175c508ac 100644
--- a/tests/decorators/test_python.py
+++ b/tests/decorators/test_python.py
@@ -256,6 +256,17 @@ class TestAirflowTaskDecorator(BasePythonTest):
 add_number()
 add_number("test")
 
+def test_fails_context_parameter_other_than_none(self):
+"""Fail if a context parameter has a default and it's not None."""
+error_message = "Context key parameter try_number can't have a default 
other than None"
+
+@task_decorator
+def add_number_to_try_number(num: int, try_number: int = 0):
+return num + try_number
+
+with pytest.raises(ValueError, match=error_message):
+add_number_to_try_number(1)
+
 def test_fail_method(self):
 """Tests that @task will fail if signature is not binding."""
 

Re: [I] Context variable defaults break tasks where these parameters don't go at the end of the task fuction definition [airflow]

[via GitHub]

potiuk closed issue #38006: Context variable defaults break tasks where these 
parameters don't go at the end of the task fuction definition
URL: https://github.com/apache/airflow/issues/38006


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

[via GitHub]

potiuk merged PR #38015:
URL: https://github.com/apache/airflow/pull/38015


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: support iam token from metadata, simplify code (#38411)

[husseinawala]

This is an automated email from the ASF dual-hosted git repository.

husseinawala pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 30817a5c6d support iam token from metadata, simplify code (#38411)
30817a5c6d is described below

commit 30817a5c6df1be3ec080ff1c542899092679768f
Author: uzhastik 
AuthorDate: Sat Mar 23 00:30:33 2024 +0300

support iam token from metadata, simplify code (#38411)
---
 airflow/providers/yandex/hooks/yq.py| 54 +++--
 airflow/providers/yandex/provider.yaml  |  3 --
 generated/provider_dependencies.json|  2 --
 pyproject.toml  |  2 --
 tests/providers/yandex/hooks/test_yq.py | 49 +-
 5 files changed, 59 insertions(+), 51 deletions(-)

diff --git a/airflow/providers/yandex/hooks/yq.py 
b/airflow/providers/yandex/hooks/yq.py
index 963709d89b..37f7550df6 100644
--- a/airflow/providers/yandex/hooks/yq.py
+++ b/airflow/providers/yandex/hooks/yq.py
@@ -16,16 +16,14 @@
 # under the License.
 from __future__ import annotations
 
-import time
 from datetime import timedelta
 from typing import Any
 
-import jwt
-import requests
-from urllib3.util.retry import Retry
+import yandexcloud
+import yandexcloud._auth_fabric as auth_fabric
+from yandex.cloud.iam.v1.iam_token_service_pb2_grpc import IamTokenServiceStub
 from yandex_query_client import YQHttpClient, YQHttpClientConfig
 
-from airflow.exceptions import AirflowException
 from airflow.providers.yandex.hooks.yandex import YandexCloudBaseHook
 from airflow.providers.yandex.utils.user_agent import provider_user_agent
 
@@ -98,35 +96,17 @@ class YQHook(YandexCloudBaseHook):
 return self.client.compose_query_web_link(query_id)
 
 def _get_iam_token(self) -> str:
-if "token" in self.credentials:
-return self.credentials["token"]
-if "service_account_key" in self.credentials:
-return 
YQHook._resolve_service_account_key(self.credentials["service_account_key"])
-raise AirflowException(f"Unknown credentials type, available keys 
{self.credentials.keys()}")
-
-@staticmethod
-def _resolve_service_account_key(sa_info: dict) -> str:
-with YQHook._create_session() as session:
-api = "https://iam.api.cloud.yandex.net/iam/v1/tokens;
-now = int(time.time())
-payload = {"aud": api, "iss": sa_info["service_account_id"], 
"iat": now, "exp": now + 360}
-
-encoded_token = jwt.encode(
-payload, sa_info["private_key"], algorithm="PS256", 
headers={"kid": sa_info["id"]}
-)
-
-data = {"jwt": encoded_token}
-iam_response = session.post(api, json=data)
-iam_response.raise_for_status()
-
-return iam_response.json()["iamToken"]
-
-@staticmethod
-def _create_session() -> requests.Session:
-session = requests.Session()
-session.verify = False
-retry = Retry(backoff_factor=0.3, total=10)
-session.mount("http://;, 
requests.adapters.HTTPAdapter(max_retries=retry))
-session.mount("https://;, 
requests.adapters.HTTPAdapter(max_retries=retry))
-
-return session
+iam_token = self.credentials.get("token")
+if iam_token is not None:
+return iam_token
+
+service_account_key = self.credentials.get("service_account_key")
+# if service_account_key is None metadata server will be used
+token_requester = 
auth_fabric.get_auth_token_requester(service_account_key=service_account_key)
+
+if service_account_key is None:
+return token_requester.get_token()
+
+sdk = yandexcloud.SDK()
+client = sdk.client(IamTokenServiceStub)
+return client.Create(token_requester.get_token_request()).iam_token
diff --git a/airflow/providers/yandex/provider.yaml 
b/airflow/providers/yandex/provider.yaml
index 0135ac3fb4..df700127c4 100644
--- a/airflow/providers/yandex/provider.yaml
+++ b/airflow/providers/yandex/provider.yaml
@@ -50,9 +50,6 @@ dependencies:
   - apache-airflow>=2.6.0
   - yandexcloud>=0.228.0
   - yandex-query-client>=0.1.2
-  - python-dateutil>=2.8.0
-  # Requests 3 if it will be released, will be heavily breaking.
-  - requests>=2.27.0,<3
 
 integrations:
   - integration-name: Yandex.Cloud
diff --git a/generated/provider_dependencies.json 
b/generated/provider_dependencies.json
index fe9848069d..4f110f9918 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -1180,8 +1180,6 @@
   "yandex": {
 "deps": [
   "apache-airflow>=2.6.0",
-  "python-dateutil>=2.8.0",
-  "requests>=2.27.0,<3",
   "yandex-query-client>=0.1.2",
   "yandexcloud>=0.228.0"
 ],
diff --git a/pyproject.toml b/pyproject.toml
index 89d7496c4a..ace1b0800a 100644
--- a/pyproject.toml
+++ 

Re: [PR] [yandexcloud/yq] support iam token from metadata, simplify code [airflow]

[via GitHub]

hussein-awala merged PR #38411:
URL: https://github.com/apache/airflow/pull/38411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

[via GitHub]

shahar1 commented on code in PR #38418:
URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715


##
airflow/providers/google/cloud/hooks/automl.py:
##
@@ -1,642 +0,0 @@
-#

Review Comment:
   Technically speaking - we didn't actually deprecate this hook, but as GCP 
disables their API, it will become unusable anyway. In Airflow's codebase it 
has been used only in the old AutoML operators, but it is also a public class - 
so people could import it if they want.
   Should we keep it with an exception raised on initialization and delete it 
later? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

[via GitHub]

shahar1 commented on code in PR #38418:
URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715


##
airflow/providers/google/cloud/hooks/automl.py:
##
@@ -1,642 +0,0 @@
-#

Review Comment:
   Technically speaking - we didn't actually deprecate this hook, but as GCP 
disables their API, it will become unusable anyway. In Airflow's codebase it 
has been used only in the old AutoML operators, but it is also a public class - 
so people could import it if they want.
   Should we keep it with an exception raised on initialization and delete it 
later, or just delete it? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Update UV to latest version released [airflow]

[via GitHub]

potiuk opened a new pull request, #38419:
URL: https://github.com/apache/airflow/pull/38419

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

[via GitHub]

shahar1 commented on code in PR #38418:
URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715


##
airflow/providers/google/cloud/hooks/automl.py:
##
@@ -1,642 +0,0 @@
-#

Review Comment:
   Technically speaking - we didn't actually deprecate this hook, but as GCP 
disables their API it will become unusable anyway.
   Should we keep it with an exception raised on initialization and delete it 
later? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Revert ObjectStorage config variables name [airflow]

[via GitHub]

potiuk merged PR #38415:
URL: https://github.com/apache/airflow/pull/38415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Revert ObjectStorage config variables name (#38415)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new fd5fe8d2c6 Revert ObjectStorage config variables name (#38415)
fd5fe8d2c6 is described below

commit fd5fe8d2c698b9c26cee47fd0af2f211e9fee7e6
Author: Pankaj Singh <98807258+pankajas...@users.noreply.github.com>
AuthorDate: Sat Mar 23 02:51:48 2024 +0530

Revert ObjectStorage config variables name (#38415)

* Revert ObjectStorage config variables name

* Fix tests
---
 airflow/providers/common/io/provider.yaml|  6 +++---
 airflow/providers/common/io/xcom/backend.py  | 12 ++--
 docs/apache-airflow-providers-common-io/xcom_backend.rst | 12 ++--
 docs/apache-airflow/core-concepts/xcoms.rst  | 12 ++--
 docs/spelling_wordlist.txt   |  1 -
 tests/providers/common/io/xcom/test_backend.py   | 10 +-
 6 files changed, 26 insertions(+), 27 deletions(-)

diff --git a/airflow/providers/common/io/provider.yaml 
b/airflow/providers/common/io/provider.yaml
index acd175c509..022205bc33 100644
--- a/airflow/providers/common/io/provider.yaml
+++ b/airflow/providers/common/io/provider.yaml
@@ -53,14 +53,14 @@ config:
   common.io:
 description: Common IO configuration section
 options:
-  xcom_objectstore_path:
+  xcom_objectstorage_path:
 description: |
   Path to a location on object storage where XComs can be stored in 
url format.
 version_added: 1.3.0
 type: string
 example: "s3://conn_id@bucket/path"
 default: ""
-  xcom_objectstore_threshold:
+  xcom_objectstorage_threshold:
 description: |
   Threshold in bytes for storing XComs in object storage. -1 means 
always store in the
   database. 0 means always store in object storage. Any positive 
number means
@@ -69,7 +69,7 @@ config:
 type: integer
 example: "100"
 default: "-1"
-  xcom_objectstore_compression:
+  xcom_objectstorage_compression:
 description: |
   Compression algorithm to use when storing XComs in object storage. 
Supported algorithms
   are a.o.: snappy, zip, gzip, bz2, and lzma. If not specified, no 
compression will be used.
diff --git a/airflow/providers/common/io/xcom/backend.py 
b/airflow/providers/common/io/xcom/backend.py
index 14d1009ead..061a90ae03 100644
--- a/airflow/providers/common/io/xcom/backend.py
+++ b/airflow/providers/common/io/xcom/backend.py
@@ -81,7 +81,7 @@ class XComObjectStoreBackend(BaseXCom):
 :raises ValueError: if the key is not relative to the configured path
 :raises TypeError: if the url is not a valid url or cannot be split
 """
-path = conf.get(SECTION, "xcom_objectstore_path", fallback="")
+path = conf.get(SECTION, "xcom_objectstorage_path", fallback="")
 p = ObjectStoragePath(path)
 
 # normalize the path
@@ -115,8 +115,8 @@ class XComObjectStoreBackend(BaseXCom):
 # we will always serialize ourselves and not by BaseXCom as the 
deserialize method
 # from BaseXCom accepts only XCom objects and not the value directly
 s_val = json.dumps(value, cls=XComEncoder).encode("utf-8")
-path = conf.get(SECTION, "xcom_objectstore_path", fallback="")
-compression = conf.get(SECTION, "xcom_objectstore_compression", 
fallback=None)
+path = conf.get(SECTION, "xcom_objectstorage_path", fallback="")
+compression = conf.get(SECTION, "xcom_objectstorage_compression", 
fallback=None)
 
 if compression:
 suffix = "." + _get_compression_suffix(compression)
@@ -124,7 +124,7 @@ class XComObjectStoreBackend(BaseXCom):
 suffix = ""
 compression = None
 
-threshold = conf.getint(SECTION, "xcom_objectstore_threshold", 
fallback=-1)
+threshold = conf.getint(SECTION, "xcom_objectstorage_threshold", 
fallback=-1)
 
 if path and -1 < threshold < len(s_val):
 # safeguard against collisions
@@ -152,7 +152,7 @@ class XComObjectStoreBackend(BaseXCom):
 Compression is inferred from the file extension.
 """
 data = BaseXCom.deserialize_value(result)
-path = conf.get(SECTION, "xcom_objectstore_path", fallback="")
+path = conf.get(SECTION, "xcom_objectstorage_path", fallback="")
 
 try:
 p = ObjectStoragePath(path) / XComObjectStoreBackend._get_key(data)
@@ -164,7 +164,7 @@ class XComObjectStoreBackend(BaseXCom):
 
 @staticmethod
 def purge(xcom: XCom, session: Session) -> None:
-path = conf.get(SECTION, "xcom_objectstore_path", fallback="")
+path = conf.get(SECTION, "xcom_objectstorage_path", fallback="")
 if isinstance(xcom.value, str):
 try:
  

[PR] Delete deprecated AutoML operators and hook [airflow]

[via GitHub]

shahar1 opened a new pull request, #38418:
URL: https://github.com/apache/airflow/pull/38418

   
   
   
   
   Starting March 31, 2024, GCP will sunset the old AutoML API, thus making the 
deprecated operators and hook effectively unusable. This PR removes all related 
logic (operators, hook and related tests).
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated (9b2d8e0fcd -> bcd7d35ff8)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 9b2d8e0fcd Upgrade to latest chart dependencies detected by canary run 
(#38416)
 add bcd7d35ff8 Update Airflow decorators stub to reflect changes in #38387 
(#38406)

No new revisions were added by this update.

Summary of changes:
 airflow/decorators/__init__.pyi | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Re: [PR] Update Airflow decorators stub to reflect changes in #38387 [airflow]

[via GitHub]

potiuk merged PR #38406:
URL: https://github.com/apache/airflow/pull/38406


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Fix `parent_model` parameter in GCP Vertex AI AutoML and Custom Job operators [airflow]

[via GitHub]

shahar1 opened a new pull request, #38417:
URL: https://github.com/apache/airflow/pull/38417

   
   
   
   
   closes: #37400
   
   The following line wrongly parsed the `parent_model` param in all Vertex AI 
AutoML "Create" operators:
   ```python
   self.parent_model = self.parent_model.rpartition("@")[0] if 
self.parent_model else None
   ```
   
   The `parent_model` should be either the actualy ID (integer) or the its 
resource name format (e.g., 
`projects/my-project/location/my-location/models/parent_model_id`). After 
removing the line, it seems to work fine (I checked it).
   
   
   ---
   
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable kubernetes_pod_operator to reattach_on_restart when the worker dies [airflow]

[via GitHub]

wircho commented on issue #21900:
URL: https://github.com/apache/airflow/issues/21900#issuecomment-2015828679

   @yeachan153 Did you ever solve this problem? We would love to be able to 
keep pods running during environment restarts, and it looks like your idea 
might work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] New generaic SQL transfer operator proposal [airflow]

[via GitHub]

potiuk commented on PR #38281:
URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015813508

   Probably yes. How to perform that?
   
   > Just the usual "Refactor -> Extract" - both are in common.sql provider, so 
adding `common.sql.util` package with `provider_compatibility.py`  and moving 
the common code there seems pretty natural


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Upgrade to latest chart dependencies detected by canary run (#38416)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 9b2d8e0fcd Upgrade to latest chart dependencies detected by canary run 
(#38416)
9b2d8e0fcd is described below

commit 9b2d8e0fcdfc4bb3efc09ad751ed5d2d7a84bea1
Author: Jarek Potiuk 
AuthorDate: Fri Mar 22 20:56:14 2024 +0100

Upgrade to latest chart dependencies detected by canary run (#38416)
---
 chart/values.schema.json | 2 +-
 chart/values.yaml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/chart/values.schema.json b/chart/values.schema.json
index 4c00d72e9e..1d132d7f10 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -668,7 +668,7 @@
 "tag": {
 "description": "The StatsD image tag.",
 "type": "string",
-"default": "v0.26.0"
+"default": "v0.26.1"
 },
 "pullPolicy": {
 "description": "The StatsD image pull policy.",
diff --git a/chart/values.yaml b/chart/values.yaml
index 29519d3e47..caa72fcc57 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -105,7 +105,7 @@ images:
 pullPolicy: IfNotPresent
   statsd:
 repository: quay.io/prometheus/statsd-exporter
-tag: v0.26.0
+tag: v0.26.1
 pullPolicy: IfNotPresent
   redis:
 repository: redis

Re: [PR] Upgrade to latest chart dependencies detected by canary run [airflow]

[via GitHub]

potiuk merged PR #38416:
URL: https://github.com/apache/airflow/pull/38416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Mysql change xcom value col type for MySQL backend [airflow]

[via GitHub]

potiuk commented on PR #38401:
URL: https://github.com/apache/airflow/pull/38401#issuecomment-2015810027

   > > I think, if that's the case then we will likely have to at least warn 
the users in significant note that it might happen and that they should likely 
run db clean on xcom table for some old data?
   > 
   > A bit Friday jokes: If take in account [last 
survey](https://docs.google.com/forms/d/1wYm6c5Gn379zkg7zD7vcWB-1fCjnOocT0oZm-tjft_Q/viewanalytics)
 - number of MySQL users about 11-14 % (11% MySQL 8, and 3.4% MySQL 5) for 
avoid long migration it could be an option to start from scratch with Postgres 
藍 .
   
   Yes. Let's also add "if mysql: sleep(0.2)" in various places in Airlfow. 
Enough to be impacting performance, but not enough to say "MYSQL does not 
work". Then we will be able to tell "But Postgres is WAY faster". 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Add experimental warning in AWS auth manager documentation (#38414)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 0d11f3ca2f Add experimental warning in AWS auth manager documentation 
(#38414)
0d11f3ca2f is described below

commit 0d11f3ca2f78f6e70e7fa9171afcfc47e5e9
Author: Vincent <97131062+vincb...@users.noreply.github.com>
AuthorDate: Fri Mar 22 16:43:03 2024 -0300

Add experimental warning in AWS auth manager documentation (#38414)
---
 docs/apache-airflow-providers-amazon/auth-manager/index.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/docs/apache-airflow-providers-amazon/auth-manager/index.rst 
b/docs/apache-airflow-providers-amazon/auth-manager/index.rst
index 0a74715894..b50918e7ef 100644
--- a/docs/apache-airflow-providers-amazon/auth-manager/index.rst
+++ b/docs/apache-airflow-providers-amazon/auth-manager/index.rst
@@ -19,6 +19,9 @@
 AWS auth manager
 
 
+.. warning::
+  The AWS auth manager is alpha/experimental at the moment and may be subject 
to change without warning.
+
 Before reading this, you should be familiar with the concept of auth manager.
 See :doc:`apache-airflow:core-concepts/auth-manager`.
 

Re: [PR] Add experimental warning in AWS auth manager documentation [airflow]

[via GitHub]

Taragolis merged PR #38414:
URL: https://github.com/apache/airflow/pull/38414


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] New generaic SQL transfer operator proposal [airflow]

[via GitHub]

frodo2000 commented on PR #38281:
URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015774537

   > How about extracting the common code for version check? Sounds reasonable 
to do it now?
   Probably yes. How to perform that?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Fix release docs for the helm chart (#38413)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 36704fb9a7 Fix release docs for the helm chart (#38413)
36704fb9a7 is described below

commit 36704fb9a7fd9fd4409057592d657d8dde542ff2
Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com>
AuthorDate: Fri Mar 22 13:15:49 2024 -0600

Fix release docs for the helm chart (#38413)
---
 dev/README_RELEASE_HELM_CHART.md | 2 +-
 dev/chart/build_changelog_annotations.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/dev/README_RELEASE_HELM_CHART.md b/dev/README_RELEASE_HELM_CHART.md
index 889f97cf9a..42e3367983 100644
--- a/dev/README_RELEASE_HELM_CHART.md
+++ b/dev/README_RELEASE_HELM_CHART.md
@@ -202,7 +202,7 @@ breeze release-management prepare-helm-chart-tarball 
--version ${VERSION} --vers
 - Generate the binary Helm Chart release:
 
 ```shell
-breeze release-management prepare-helm-chart-packagte --sig 
jedcunning...@apache.org
+breeze release-management prepare-helm-chart-package --sign-email 
jedcunning...@apache.org
 ```
 
 Warning: you need the `helm gpg` plugin to sign the chart (instructions to 
install it above)
diff --git a/dev/chart/build_changelog_annotations.py 
b/dev/chart/build_changelog_annotations.py
index 0a7e57844d..efc48857db 100755
--- a/dev/chart/build_changelog_annotations.py
+++ b/dev/chart/build_changelog_annotations.py
@@ -98,7 +98,7 @@ with open("chart/RELEASE_NOTES.rst") as f:
 in_first_release = True
 # Make sure we get past "significant features" before we actually 
start keeping track
 elif not past_significant_changes:
-if line == "New Features":
+if line in TYPE_MAPPING:
 section = line
 past_significant_changes = True
 elif not line.startswith("- "):

Re: [PR] Fix release docs for the helm chart [airflow]

[via GitHub]

potiuk merged PR #38413:
URL: https://github.com/apache/airflow/pull/38413


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Upgrade to latest chart dependencies detected by canary run [airflow]

[via GitHub]

potiuk opened a new pull request, #38416:
URL: https://github.com/apache/airflow/pull/38416

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fix XComObjectStoreBackend config var in docs and bug when xcom_objectstore_compression is empty [airflow]

[via GitHub]

pankajastro commented on PR #38142:
URL: https://github.com/apache/airflow/pull/38142#issuecomment-2015737146

   @bolkedebruin create a PR https://github.com/apache/airflow/pull/38415 PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Revert ObjectStorage config variables name [airflow]

[via GitHub]

pankajastro opened a new pull request, #38415:
URL: https://github.com/apache/airflow/pull/38415

   Address https://github.com/apache/airflow/pull/38142#issuecomment-2007166415
   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Migrate to connexion v3 [airflow]

[via GitHub]

potiuk commented on PR #37638:
URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015695912

   > ### Does not raise Valueerror
   
   When I run it in my unit test, I got Internal Server error - that's why it 
does not return ValueError, because it converts ValueError into 500 internal 
HTTP error:
   
   ```
   FAILED [100%][2024-03-22T19:17:02.835+0100] {exceptions.py:97} ERROR - 
ValueError("'dummy' is not a valid DagRunState")
   Traceback (most recent call last):
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/_exception_handler.py",
 line 53, in wrapped_app
   await app(scope, receive, sender)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/swagger_ui.py",
 line 222, in __call__
   await self.router(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/routing.py",
 line 756, in __call__
   await self.middleware_stack(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/routing.py",
 line 806, in app
   await self.default(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/swagger_ui.py",
 line 235, in default_fn
   await self.app(original_scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/middleware/cors.py",
 line 85, in __call__
   await self.app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/routing.py",
 line 154, in __call__
   await self.router(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/routing.py",
 line 756, in __call__
   await self.middleware_stack(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/starlette/routing.py",
 line 806, in app
   await self.default(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/routing.py",
 line 48, in __call__
   await self.next_app(original_scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/abstract.py",
 line 268, in __call__
   await self.app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/abstract.py",
 line 268, in __call__
   await self.app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/abstract.py",
 line 268, in __call__
   await self.app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/lifespan.py",
 line 26, in __call__
   await self.next_app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/middleware/abstract.py",
 line 268, in __call__
   await self.app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/connexion/apps/flask.py",
 line 151, in __call__
   return await self.asgi_app(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/a2wsgi/wsgi.py",
 line 165, in __call__
   return await responder(scope, receive, send)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/a2wsgi/wsgi.py",
 line 200, in __call__
   await self.loop.run_in_executor(
 File 
"/home/jarek/.local/share/hatch/pythons/3.8/python/lib/python3.8/concurrent/futures/thread.py",
 line 57, in run
   result = self.fn(*self.args, **self.kwargs)
 File 
"/home/jarek/.local/share/hatch/env/virtual/apache-airflow/AdTkC0pY/apache-airflow/lib/python3.8/site-packages/a2wsgi/wsgi.py",
 line 256, in wsgi
   iterable = self.app(environ, start_response)
 File 

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

[via GitHub]

ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015691919

   > > Nice. The prefix is cool.
   > > There is one worry with that U just wanted to mention it.
   > > I understand that we are masking secreates before serialization ? Just 
wanted to amke sure sure of it - it looks like from the code , but I wanted to 
make sure we are not showing half-unmasked secrets accidentally when we 
truncate the secrets.
   > 
   > The secrets are masked after serialization see:
   > 
   > 
https://github.com/apache/airflow/blob/6a225ccb2a9b51c52df88368323993bcd017be0d/airflow/models/renderedtifields.py#L127
   > 
   > . I have added a test and will also do manual testing
   
   It doesn't work so


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

[via GitHub]

ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015670623

   > Nice. The prefix is cool.
   > 
   > There is one worry with that U just wanted to mention it.
   > 
   > I understand that we are masking secreates before serialization ? Just 
wanted to amke sure sure of it - it looks like from the code , but I wanted to 
make sure we are not showing half-unmasked secrets accidentally when we 
truncate the secrets.
   
   The secrets are masked after serialization see: 
https://github.com/apache/airflow/blob/6a225ccb2a9b51c52df88368323993bcd017be0d/airflow/models/renderedtifields.py#L127.
 I have added a test and will also do manual testing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add declarative configuration to define custom roles [airflow]

[via GitHub]

ketozhang commented on issue #38347:
URL: https://github.com/apache/airflow/issues/38347#issuecomment-2015651861

   Thanks and yep, it's a classic problem most devops folks have to deal with. 
It's heavily dependent if you want to lock down configs post-deploy and if DB 
should be overwritten during a redeploy.
   
   What's currently being done for connections and variables to solve that 
issue? They both already have declarative equivalents.
   
   > So far all the efforts failed on that step
   
   How have they failed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

[via GitHub]

potiuk commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2015639112

   No. I do not see anyone working on it nor providing a solution.  But the log 
above suggests that this is caused by so called "mini scheduler" which is 
enabled by this configuration 
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution
 - likely disabling it will mitigate the problem (at the expense of slightly 
increased latency between tasks.
   
   Can everyone in this thread make an experiment, disable it and report back 
here - at least this will give some clue on where the problems might happen.
   
   cc: @kaxil @ephraimbuddy -> problems similar to that one keep on appearing, 
I think there is some kind of deadlock/resource contention with mini scheduler 
in some specific cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

[via GitHub]

potiuk merged PR #38412:
URL: https://github.com/apache/airflow/pull/38412


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Make postgresql default engine args comply with SA 2.0 [airflow]

[via GitHub]

potiuk merged PR #38362:
URL: https://github.com/apache/airflow/pull/38362


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Migrate to connexion v3 [airflow]

[via GitHub]

sudiptob2 commented on PR #37638:
URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015632855

   > Since is it API experimental (which is deprecated), I would not spend too 
much time on it. Maybe change the expected exception? Change it to expect 
`ValueError` from `connexion` library
   
   Actually, I did tried that. That exception is not propagated to the top 
level function we are testing. Therefore, inside the test, we can not catch any 
`exception`. 
   
   I will move ahead with other tests for now.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: to avoid type's mismatch durind concatenation modified to f-strings (#38412)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 6a225ccb2a to avoid type's mismatch durind concatenation modified to 
f-strings (#38412)
6a225ccb2a is described below

commit 6a225ccb2a9b51c52df88368323993bcd017be0d
Author: igeni 
AuthorDate: Fri Mar 22 20:52:14 2024 +0300

to avoid type's mismatch durind concatenation modified to f-strings (#38412)
---
 airflow/cli/commands/info_command.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/airflow/cli/commands/info_command.py 
b/airflow/cli/commands/info_command.py
index 2b4963179e..26e2b37bcc 100644
--- a/airflow/cli/commands/info_command.py
+++ b/airflow/cli/commands/info_command.py
@@ -84,7 +84,7 @@ class PiiAnonymizer(Anonymizer):
 def process_username(self, value) -> str:
 if not value:
 return value
-return value[0] + "..." + value[-1]
+return f"{value[0]}...{value[-1]}"
 
 def process_url(self, value) -> str:
 if not value:
@@ -114,11 +114,11 @@ class PiiAnonymizer(Anonymizer):
 
 # pack
 if username and password and host:
-netloc = username + ":" + password + "@" + host
+netloc = f"{username}:{password}@{host}"
 elif username and host:
-netloc = username + "@" + host
+netloc = f"{username}@{host}"
 elif password and host:
-netloc = ":" + password + "@" + host
+netloc = f":{password}@{host}"
 elif host:
 netloc = host
 else:
@@ -217,7 +217,7 @@ class AirflowInfo:
 if module is None or module == str.__class__.__module__:
 return o.__class__.__name__  # Avoid reporting __builtin__
 else:
-return module + "." + o.__class__.__name__
+return f"{module}.{o.__class__.__name__}"
 
 try:
 handler_names = [get_fullname(handler) for handler in 
logging.getLogger("airflow.task").handlers]

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

[via GitHub]

boring-cyborg[bot] commented on PR #38412:
URL: https://github.com/apache/airflow/pull/38412#issuecomment-2015621212

   Awesome work, congrats on your first merged pull request! You are invited to 
check our [Issue Tracker](https://github.com/apache/airflow/issues) for 
additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Make postgresql default engine args comply with SA 2.0 (#38362)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 5344881715 Make postgresql default engine args comply with SA 2.0 
(#38362)
5344881715 is described below

commit 5344881715156b98521706b2facb261a499f2e4b
Author: Andrey Anshin 
AuthorDate: Fri Mar 22 21:51:27 2024 +0400

Make postgresql default engine args comply with SA 2.0 (#38362)

* Change Postgres default executemany_mode to 'values_plus_batch'

* executemany_values_page_size > insertmanyvalues_page_size for SA20
---
 airflow/settings.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/airflow/settings.py b/airflow/settings.py
index 6cbf6126e0..2a2b1455bc 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -273,8 +273,8 @@ def configure_orm(disable_connection_pool=False, 
pool_class=None):
 
 DEFAULT_ENGINE_ARGS = {
 "postgresql": {
-"executemany_mode": "values",
-"executemany_values_page_size": 1,
+"executemany_mode": "values_plus_batch",
+"executemany_values_page_size" if is_sqlalchemy_v1() else 
"insertmanyvalues_page_size": 1,
 "executemany_batch_page_size": 2000,
 },
 }

Re: [I] Airflow postgres executemany_mode': 'values' error [airflow]

[via GitHub]

potiuk closed issue #38064: Airflow postgres executemany_mode': 'values' error
URL: https://github.com/apache/airflow/issues/38064


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fix release docs for the helm chart [airflow]

[via GitHub]

potiuk commented on code in PR #38413:
URL: https://github.com/apache/airflow/pull/38413#discussion_r1535975928


##
dev/chart/build_changelog_annotations.py:
##
@@ -98,7 +98,7 @@ def print_entry(section: str, description: str, pr_number: 
int | None):
 in_first_release = True
 # Make sure we get past "significant features" before we actually 
start keeping track
 elif not past_significant_changes:
-if line == "New Features":
+if line in TYPE_MAPPING:

Review Comment:
   It's quite obvious - the code will fail if there are no New Features (like 
in 1.13.1 



##
dev/chart/build_changelog_annotations.py:
##
@@ -98,7 +98,7 @@ def print_entry(section: str, description: str, pr_number: 
int | None):
 in_first_release = True
 # Make sure we get past "significant features" before we actually 
start keeping track
 elif not past_significant_changes:
-if line == "New Features":
+if line in TYPE_MAPPING:

Review Comment:
   It's quite obvious - the old code will fail if there are no New Features 
(like in 1.13.1 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fix release docs for the helm chart [airflow]

[via GitHub]

enchant3dmango commented on code in PR #38413:
URL: https://github.com/apache/airflow/pull/38413#discussion_r1535967153


##
dev/chart/build_changelog_annotations.py:
##
@@ -98,7 +98,7 @@ def print_entry(section: str, description: str, pr_number: 
int | None):
 in_first_release = True
 # Make sure we get past "significant features" before we actually 
start keeping track
 elif not past_significant_changes:
-if line == "New Features":
+if line in TYPE_MAPPING:

Review Comment:
   The TYPE_MAPPING has 5 key-value pairs in it.
   
   So this change might have a different result from the previous code which 
only checks if the line equals "New Features". Could you explain this change, 
please?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Status of testing of Apache Airflow Helm Chart 1.13.1rc1 [airflow]

[via GitHub]

brokenjacobs commented on issue #38382:
URL: https://github.com/apache/airflow/issues/38382#issuecomment-2015577073

   LGTM!, keda bypass of pgbouncer is working as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated (947c48b2fb -> 40559a315e)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 947c48b2fb DockerOperator: use DOCKER_HOST as default for docker_url 
(#38387)
 add 40559a315e Fix typo in verification process for Helm Chart (#38410)

No new revisions were added by this update.

Summary of changes:
 dev/README_RELEASE_HELM_CHART.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Re: [PR] Fix typo in verification process for Helm Chart [airflow]

[via GitHub]

potiuk merged PR #38410:
URL: https://github.com/apache/airflow/pull/38410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add declarative configuration to define custom roles [airflow]

[via GitHub]

potiuk commented on issue #38347:
URL: https://github.com/apache/airflow/issues/38347#issuecomment-2015572014

   Sure. If you have a good idea how to do it in a robust way - proposing a 
solution, especially one that will not overwrite existing DB and will allow to 
both declaratively upload roles from a source and manually modify them, and 
keep those local modificiattions - PRs are most welcome.
   
   Just make sure when you submit the PR to propose a solution for this 
conundrum where you have both manually modifiable data and uploadable from the 
externally depoyable sources in declarative way. So far all the efforts failed 
on that step, but maybe you will be able to propose something. 
   
   BTW. once you find a solution, also similar thing should be done from 
connections, users and variables - those suffer from the same issue and if you 
find and get approval to solve it for roles, applying similar approach for the 
others is a natural next step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Migrate to connexion v3 [airflow]

[via GitHub]

vincbeck commented on PR #37638:
URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015557108

   > ### Does not raise Valueerror
   > This test seems not to raise `ValueError` anymore after migrating to 
connexion v3. See **screenshot 1**. But a ValueError is indeed raised from deep 
inside the `Connexion` library. Need some context on what this test is testing. 
Also suggestion on possible rewrite.
   > 
   > 
https://github.com/apache/airflow/blob/947c48b2fbfaed59068e4a06e223cf501cc1227f/tests/www/api/experimental/test_dag_runs_endpoint.py#L98-L106
   > 
   > **Screenshot 1**
   > 
   > https://private-user-images.githubusercontent.com/32765701/316107238-40cd380e-0939-4851-8e43-279ed6f63e95.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTExMjgyOTcsIm5iZiI6MTcxMTEyNzk5NywicGF0aCI6Ii8zMjc2NTcwMS8zMTYxMDcyMzgtNDBjZDM4MGUtMDkzOS00ODUxLThlNDMtMjc5ZWQ2ZjYzZTk1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzIyVDE3MTk1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNkOGQ4OTIxZjFhNjY3Yjg3NjNmMTgzOTkyZWNlYzVhMTVjMDYyNWY1ZmM5ZDU3NmExZTQwNTMyZDM5NGRhYTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.KUFWFBOmodbKxTQ1z-axFEmr7MH7Sa8srFahChmXxHQ;>
   > **An exception occurred deep inside the library but was not propagated.**
   > 
   > https://private-user-images.githubusercontent.com/32765701/316107584-ec602400-a53e-4e6a-8b11-656b0be413c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTExMjgyOTcsIm5iZiI6MTcxMTEyNzk5NywicGF0aCI6Ii8zMjc2NTcwMS8zMTYxMDc1ODQtZWM2MDI0MDAtYTUzZS00ZTZhLThiMTEtNjU2YjBiZTQxM2MzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzIyVDE3MTk1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWUxOTk1NWNmMTU3Mjg5NzkwZjM1YzNkMTJhOGM5NmU5OGZkNmEwMTcxNDRkNzM3NmNhMGE3MjE2NGJhNDRkZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.4Aip4eE8wSnsGCi6ev8ANzZm_cyGUa0PoIOIrK_Lk98;>
   
   Since is it API experimental, I would not spend too much time on it. Maybe 
change the expected exception? Change it to expect `ValueError` from 
`connexion` library


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Status of testing of Apache Airflow Helm Chart 1.13.1rc1 [airflow]

[via GitHub]

brokenjacobs commented on issue #38382:
URL: https://github.com/apache/airflow/issues/38382#issuecomment-2015552750

   quay.io outage causing issues with the statsd container:
   
![image](https://github.com/apache/airflow/assets/3159463/562b91ea-2813-435e-80fb-396f489c00f2)
   ```
 Warning  Failed 15s kubelet
Failed to pull image "quay.io/prometheus/statsd-exporter:v0.26.0": failed 
to pull and unpack image "quay.io/prometheus/statsd-exporter:v0.26.0": failed 
to copy: httpReadSeeker: failed open: unexpected status code 
https://quay.io/v2/prometheus/statsd-exporter/blobs/sha256:4f76ba478baf84e4e9091fe66b1e53b70d9f1dfa1bd0ce24fcd4aec18e0b3929:
 502 Bad Gateway
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Add experimental warning on AWS auth manager documentation [airflow]

[via GitHub]

vincbeck opened a new pull request, #38414:
URL: https://github.com/apache/airflow/pull/38414

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fix pre-mature evaluation of tasks in mapped task group [airflow]

[via GitHub]

ephraimbuddy commented on PR #36462:
URL: https://github.com/apache/airflow/pull/36462#issuecomment-2015546025

   Can you add this test:
   ```
   def 
test_mapped_tasks_in_mapped_task_group_waits_for_upstreams_to_complete(dag_maker,
 session):
   """Test that one failed trigger rule works well in mapped task group"""
   with dag_maker() as dag:
   
   @dag.task
   def t1():
   return [1, 2, 3]
   
   @task_group("tg1")
   def tg1(a):
   @dag.task()
   def t2(a):
   return a
   
   @dag.task(trigger_rule=TriggerRule.ONE_FAILED)
   def t3(a):
   return a
   
   t2(a) >> t3(a)
   
   t = t1()
   tg1.expand(a=t)
   
   dr = dag_maker.create_dagrun()
   ti = dr.get_task_instance(task_id="t1")
   ti.run()
   dr.task_instance_scheduling_decisions()
   ti3 = dr.get_task_instance(task_id="tg1.t3")
   assert not ti3.state
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Fix release docs for the helm chart [airflow]

[via GitHub]

jedcunningham opened a new pull request, #38413:
URL: https://github.com/apache/airflow/pull/38413

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] New generaic SQL transfer operator proposal [airflow]

[via GitHub]

potiuk commented on PR #38281:
URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015543690

   How about extracting the common code for version check? Sounds reasonable to 
do it now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

[via GitHub]

igeni opened a new pull request, #38412:
URL: https://github.com/apache/airflow/pull/38412

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

[via GitHub]

boring-cyborg[bot] commented on PR #38412:
URL: https://github.com/apache/airflow/pull/38412#issuecomment-2015543056

   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contributors' Guide 
(https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst)
 for testing locally, it's a heavy docker but it ships with a working Airflow 
and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail 
due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it 
better .
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] New generaic SQL transfer operator proposal [airflow]

[via GitHub]

potiuk commented on code in PR #38281:
URL: https://github.com/apache/airflow/pull/38281#discussion_r1535923249


##
airflow/providers/common/sql/transfers/sql.py:
##
@@ -0,0 +1,248 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import re
+from contextlib import closing
+from typing import TYPE_CHECKING, Any, Iterable, Mapping, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.models import BaseOperator
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+from airflow.utils.context import Context
+
+_PROVIDERS_MATCHER = re.compile(r"airflow\.providers\.(.*)\.hooks.*")
+
+_MIN_SUPPORTED_PROVIDERS_VERSION = {
+"amazon": "4.1.0",
+"apache.drill": "2.1.0",
+"apache.druid": "3.1.0",
+"apache.hive": "3.1.0",
+"apache.pinot": "3.1.0",
+"databricks": "3.1.0",
+"elasticsearch": "4.1.0",
+"exasol": "3.1.0",
+"google": "8.2.0",
+"jdbc": "3.1.0",
+"mssql": "3.1.0",
+"mysql": "3.1.0",
+"odbc": "3.1.0",
+"oracle": "3.1.0",
+"postgres": "5.1.0",
+"presto": "3.1.0",
+"qubole": "3.1.0",
+"slack": "5.1.0",
+"snowflake": "3.1.0",
+"sqlite": "3.1.0",
+"trino": "3.1.0",
+"vertica": "3.1.0",
+}
+
+
+class SqlToSqlOperator(BaseOperator):
+"""
+Copy sql output data from one base to another database table.

Review Comment:
   Yep.qubole has been removed from maintenance last month indeed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Check task attribute before use in sentry.add_tagging() [airflow]

[via GitHub]

ephraimbuddy commented on code in PR #37143:
URL: https://github.com/apache/airflow/pull/37143#discussion_r1535920258


##
airflow/sentry/__init__.py:
##
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Sentry Integration."""
+
+from __future__ import annotations
+
+from airflow.configuration import conf
+from airflow.sentry.blank import BlankSentry
+
+Sentry: BlankSentry = BlankSentry()
+d = conf.getboolean("sentry", "sentry_on", fallback=False)

Review Comment:
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [yandexcloud/yq] support iam token from metadata, simplify code [airflow]

[via GitHub]

uzhastik opened a new pull request, #38411:
URL: https://github.com/apache/airflow/pull/38411

   
   
   
   
   
   Add ability to use IAM token from metadata service in YQ operator
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

[via GitHub]

jedcunningham commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535918174


##
airflow/serialization/helpers.py:
##
@@ -38,6 +39,13 @@ def is_jsonable(x):
 else:
 return True
 
+max_length = conf.getint("core", "max_templated_field_length")
+
+if template_field and len(str(template_field)) > max_length:
+return (
+f"{str(template_field)[:max_length-79]}... truncated. "
+"You can change this behaviour in [core]max_templated_field_length"

Review Comment:
   Let's do this as a prefix instead. It's more obvious if its at the beginning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] New generaic SQL transfer operator proposal [airflow]

[via GitHub]

frodo2000 commented on code in PR #38281:
URL: https://github.com/apache/airflow/pull/38281#discussion_r1535918264


##
airflow/providers/common/sql/transfers/sql.py:
##
@@ -0,0 +1,248 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import re
+from contextlib import closing
+from typing import TYPE_CHECKING, Any, Iterable, Mapping, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.models import BaseOperator
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+from airflow.utils.context import Context
+
+_PROVIDERS_MATCHER = re.compile(r"airflow\.providers\.(.*)\.hooks.*")
+
+_MIN_SUPPORTED_PROVIDERS_VERSION = {
+"amazon": "4.1.0",
+"apache.drill": "2.1.0",
+"apache.druid": "3.1.0",
+"apache.hive": "3.1.0",
+"apache.pinot": "3.1.0",
+"databricks": "3.1.0",
+"elasticsearch": "4.1.0",
+"exasol": "3.1.0",
+"google": "8.2.0",
+"jdbc": "3.1.0",
+"mssql": "3.1.0",
+"mysql": "3.1.0",
+"odbc": "3.1.0",
+"oracle": "3.1.0",
+"postgres": "5.1.0",
+"presto": "3.1.0",
+"qubole": "3.1.0",
+"slack": "5.1.0",
+"snowflake": "3.1.0",
+"sqlite": "3.1.0",
+"trino": "3.1.0",
+"vertica": "3.1.0",
+}
+
+
+class SqlToSqlOperator(BaseOperator):
+"""
+Copy sql output data from one base to another database table.

Review Comment:
   slack and qubole removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Fix typo in verification process for Helm Chart [airflow]

[via GitHub]

potiuk opened a new pull request, #38410:
URL: https://github.com/apache/airflow/pull/38410

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Check task attribute before use in sentry.add_tagging() [airflow]

[via GitHub]

LipuFei commented on PR #37143:
URL: https://github.com/apache/airflow/pull/37143#issuecomment-2015525725

   @potiuk I will split the commits so they are easier to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Migrate to connexion v3 [airflow]

[via GitHub]

sudiptob2 commented on PR #37638:
URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015521624

   ### Does not raise Valueerror
   This test seems not to raise `ValueError` anymore after migrating to 
connexion v3. See **screenshot 1**.  But a ValueError is indeed raised from 
deep inside the `Connexion` library. Need some context on what this test is 
testing. Also suggestion on possible rewrite.
   
https://github.com/apache/airflow/blob/947c48b2fbfaed59068e4a06e223cf501cc1227f/tests/www/api/experimental/test_dag_runs_endpoint.py#L98-L106
   
   **Screenshot 1**
   
   https://github.com/apache/airflow/assets/32765701/40cd380e-0939-4851-8e43-279ed6f63e95;>
   
   
   **An exception occurred deep inside the library but was not propagated.**
   
   https://github.com/apache/airflow/assets/32765701/ec602400-a53e-4e6a-8b11-656b0be413c3;>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] feat: Add DbtCloudRetryJobOperator to retry failed dbt job [airflow]

[via GitHub]

andyguwc commented on PR #38001:
URL: https://github.com/apache/airflow/pull/38001#issuecomment-2015519361

   > Why not create a flag on the DbtCloudRunJobOperator to retry from failure 
rather than create a new operator that reuses a lot of the same code?
   
   @josh-fell thanks. I was actually going back and forth between these 
approaches. I thought separating is cleaner because the operator only does one 
thing, and the user doesn't need to think through both "retry" from airflow 
task perspective and "retry" by calling that DBT API to retry from point of 
failure. 
   
   Let me think more about this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org