[GitHub] [airflow] potiuk commented on pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk commented on PR #29888:
URL: https://github.com/apache/airflow/pull/29888#issuecomment-1453119981

   inlined all the constants now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk commented on PR #29888:
URL: https://github.com/apache/airflow/pull/29888#issuecomment-1453118275

   > I wonder if we should just move all the underscore variables.
   
   Yeah. I was on the fence with this one, but I realized this is far more than 
our test's problem. Right now if anyone uses old google provider with the new 
google auth, it will miserably fail, so the risk of having the imports are far 
bigger than inlining all the imports there. cc: @eladkal - this one SHOULD be 
included in the new wave of providers, as soon we might be flooded with "google 
provider stopped working" and we will have no other workarounds for those users 
but "downgrade google-auth package  to 2.16.1. When we release this in the 
provider, we will also be able to tell them "upgrade google provider".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk closed pull request #29888: Move help message to the google auth code
URL: https://github.com/apache/airflow/pull/29888


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] shahar1 commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator

2023-03-02 Thread via GitHub 


shahar1 commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-1453096326

   I've been trying to resolve this issue, and it seems that:
   - `GKEStartPodOperator` currently supports an impersonation chain. (#19518)
   - `DataflowCreateJavaJobOperator` and `DataflowCreatePythonJobOperator` are 
both deprecated in favor of `BeamRunPythonPipelineOperator`. (#12814)
   - `BeamRunPythonPipelineOperator` supports the impersonation chain by 
passing `DataflowConfiguration` that includes `impersonation_chain` parameter. 
(#14094)
   
   Therefore, this issue could be closed as it is already resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell opened a new pull request, #29892: Add testing a connection via Airflow CLI

2023-03-02 Thread via GitHub 


josh-fell opened a new pull request, #29892:
URL: https://github.com/apache/airflow/pull/29892

   Closes: #29875
   
   This PR introduces a new Airflow CLI command -- `airflow connections test 
` -- to test a single, predefined Airflow connection.
   
   _Successful test_
   https://user-images.githubusercontent.com/48934154/222654162-998fbe8e-e1c0-4ec1-846e-896f34067480.png;>
   
   _Failed test_
   https://user-images.githubusercontent.com/48934154/222654246-043df163-a3e0-4d86-87fc-e780fe13572f.png;>
   
   
   _TODO:_
   - [ ] Add tests (unironically)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29832: Add `_on_failure_fail_dagrun` attribute for teardown tasks

2023-03-02 Thread via GitHub 


ephraimbuddy commented on code in PR #29832:
URL: https://github.com/apache/airflow/pull/29832#discussion_r1124058690


##
airflow/utils/task_group.py:
##
@@ -245,6 +245,8 @@ def add(self, task: DAGNode) -> None:
 elif SetupTeardownContext.is_teardown:
 if isinstance(task, AbstractOperator):
 setattr(task, "_is_teardown", True)
+if getattr(SetupTeardownContext, "on_failure_fail_dagrun", 
False):
+setattr(task, "_on_failure_fail_dagrun", True)

Review Comment:
   If that's the case, should we still have @setup/@teardown decorator because 
it seems to me that users would now be able to set their tasks as 
setup/teardown when defining it in the task? e.g:
   ```python
   
   BashOperator(task_id='mytask', is_setup=True, bash_command="echo 1")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy opened a new pull request, #29891: Classic TaskGroup setup/teardown

2023-03-02 Thread via GitHub 


ephraimbuddy opened a new pull request, #29891:
URL: https://github.com/apache/airflow/pull/29891

   This implements the classic TaskGroup setup/teardown. Ensures that nested 
TaskGroups are taken into account
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #29850: Fixing broken filter in in /taskinstance/list view

2023-03-02 Thread via GitHub 


uranusjr commented on code in PR #29850:
URL: https://github.com/apache/airflow/pull/29850#discussion_r1124054180


##
airflow/models/taskinstance.py:
##
@@ -139,6 +139,10 @@
 from airflow.models.operator import Operator
 from airflow.utils.task_group import MappedTaskGroup, TaskGroup
 
+hybrid_property = property
+else:
+from sqlalchemy.ext.hybrid import hybrid_property

Review Comment:
   Can you add a TODO comment referencing the issue? This makes it easier to 
review and potentially remove the workwround in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #29815: Add retry to the scheduler loop to protect against DB hiccups

2023-03-02 Thread via GitHub 


uranusjr commented on PR #29815:
URL: https://github.com/apache/airflow/pull/29815#issuecomment-1453013130

   There are multiple db commits in the block and I wonder if the internal 
state would be messed up if the db fails midway through and the entire block 
got rerun.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #29689: Add location to CloudBuildCreateBuildOperator

2023-03-02 Thread via GitHub 


uranusjr commented on code in PR #29689:
URL: https://github.com/apache/airflow/pull/29689#discussion_r1124049541


##
airflow/providers/google/cloud/hooks/cloud_build.py:
##
@@ -158,13 +159,17 @@ def create_build_without_waiting_for_result(
 :param timeout: Optional, the amount of time, in seconds, to wait for 
the request to complete.
 Note that if `retry` is specified, the timeout applies to each 
individual attempt.
 :param metadata: Optional, additional metadata that is provided to the 
method.
+:param location: Optional, The location of the project.
+If set to None or missing, global is used.

Review Comment:
   What does “global” mean here? It may be worthwhile to clarify in the 
docstring. (Or maybe I’m just missing context here, I’m not very familiar with 
the Google Cloud Platform.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #29842: Allow for templated args to be passed to the result_processor function in the GithubSensor fixing templated tags in GithubTagSensor

2023-03-02 Thread via GitHub 


uranusjr commented on PR #29842:
URL: https://github.com/apache/airflow/pull/29842#issuecomment-1453008723

   THe breakage sounds reasonable if we bump the GitHub provider to 3.0.0. So 
the question would be, is having the kwarg default to true a good idea? Let’s 
bump if it is.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #29840: DatabricksSubmitRunOperator to support taskflow

2023-03-02 Thread via GitHub 


uranusjr commented on code in PR #29840:
URL: https://github.com/apache/airflow/pull/29840#discussion_r1124046647


##
airflow/providers/databricks/operators/databricks.py:
##
@@ -285,12 +285,12 @@ def __init__(
 *,
 json: Any | None = None,
 tasks: list[object] | None = None,
-spark_jar_task: dict[str, str] | None = None,
-notebook_task: dict[str, str] | None = None,
-spark_python_task: dict[str, str | list[str]] | None = None,
-spark_submit_task: dict[str, list[str]] | None = None,
-pipeline_task: dict[str, str] | None = None,
-dbt_task: dict[str, str | list[str]] | None = None,

Review Comment:
   I may also be a good idea to split the changes into two PRs, one does not 
need to block the other.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29890: Import error for dagfactory module

2023-03-02 Thread via GitHub 


boring-cyborg[bot] commented on issue #29890:
URL: https://github.com/apache/airflow/issues/29890#issuecomment-1453001049

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Narsimha1997 opened a new issue, #29890: Import error for dagfactory module

2023-03-02 Thread via GitHub 


Narsimha1997 opened a new issue, #29890:
URL: https://github.com/apache/airflow/issues/29890

   ### Apache Airflow version
   
   2.5.1
   
   ### What happened
   
   Iam running airflow version 2.3.0 through docker and python version is 
3.10.6, i installed dag-factory python module using pip and but getting error 
in the airflow web interface showing that 'No module named dagfactory'. I also 
entered into container and again installed the module using pip but same same 
issue, any suggestions please let me know.
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   you can mail to me.
   
   ### Operating System
   
   Ubutnu
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] abin-tiger commented on issue #28751: KubernetesExecutor leaves failed pods due to deepcopy issue with Google providers

2023-03-02 Thread via GitHub 


abin-tiger commented on issue #28751:
URL: https://github.com/apache/airflow/issues/28751#issuecomment-1452996944

   We're facing the same issue when using KubernetesExecutor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ftaib commented on issue #27545: missing volumes for git-sync-ssh-key

2023-03-02 Thread via GitHub 


ftaib commented on issue #27545:
URL: https://github.com/apache/airflow/issues/27545#issuecomment-1452938150

   When will the new Helm Chart be available with this changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ftaib commented on issue #27476: Volume is missing for sshKeySecret when dag persistence is enabled.

2023-03-02 Thread via GitHub 


ftaib commented on issue #27476:
URL: https://github.com/apache/airflow/issues/27476#issuecomment-1452930037

   Hi, may I know when will the new Helm Chart will be available with the 
changes above?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


uranusjr commented on PR #29888:
URL: https://github.com/apache/airflow/pull/29888#issuecomment-1452928059

   I wonder if we should just move all the underscore variables.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #29609: Decouple production executor warning in dags UI

2023-03-02 Thread via GitHub 


uranusjr commented on PR #29609:
URL: https://github.com/apache/airflow/pull/29609#issuecomment-1452912263

   Scraping it sounds reasonable to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell commented on a diff in pull request #29694: `GoogleDriveHook`: Fixing log message + adding more verbose documentation

2023-03-02 Thread via GitHub 


josh-fell commented on code in PR #29694:
URL: https://github.com/apache/airflow/pull/29694#discussion_r1123963291


##
airflow/providers/google/suite/hooks/drive.py:
##
@@ -243,8 +300,21 @@ def upload_file(
 .create(body=file_metadata, media_body=media, fields="id", 
supportsAllDrives=True)
 .execute(num_retries=self.num_retries)
 )
-self.log.info("File %s uploaded to gdrive://%s.", local_location, 
remote_location)
-return file.get("id")
+file_id = file.get("id")
+
+upload_location = remote_location
+
+if folder_id != "root":
+try:
+upload_location = self._resolve_file_path(folder_id)
+except (GoogleApiClientError, AirflowException) as e:

Review Comment:
   I agree with @aru-trackunit. I don't think the task should fail simply 
because the file path can't be resolved for logging purposes. It's unfortunate 
the Google API doesn't surface this information easily. As long as the task 
does what it's supposed to do, Airflow shouldn't care what is in its 
user-facing task logs. Having this logic configurable also helps too.
   
   I do agree with @eladkal though that `AirflowException` is irrelevant. The 
operator is not failing for some orchestration or execution-related issue (IMO 
I think we use `AirflowException` with too broad of a brush sometimes). Maybe a 
custom exception is raised (e.g. `NestedFolderLimitReachedException` or 
something similar) just in case some real AirflowException is swallowed 
unknowingly?



##
airflow/providers/google/suite/hooks/drive.py:
##
@@ -159,6 +161,59 @@ def exists(
 )
 )
 
+def _get_file_info(self, file_id: str):
+"""
+Returns Google API file_info object containing id, name, parents in 
the response
+https://developers.google.com/drive/api/v3/reference/files/get
+
+:param file_id: id as string representation of interested file
+:return: file
+"""
+file_info = (
+self.get_conn()
+.files()
+.get(
+fileId=file_id,
+fields="id,name,parents",
+supportsAllDrives=True,
+)
+.execute(num_retries=2)
+)
+return file_info
+
+def _resolve_file_path(self, file_id: str) -> str:
+"""
+Returns the full Google Drive path for given file_id
+
+:param file_id: The id of a file in Google Drive
+:return: Google Drive full path for a file
+"""
+MAX_NESTED_FOLDERS_LEVEL = 20  # Link to docs 
https://support.google.com/a/users/answer/7338880?hl=en

Review Comment:
   This still feels magical. But, out of curiosity, if the documentation states 
"A folder in a shared drive can have up to 20 levels of nested folders", does 
that mean users _can't_ physically create more than that number of nested 
folders or is it really a _shouldn't_ create situation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] agentdanger commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


agentdanger commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452826643

   I was able to solve this issue by installing 1.8 with the configured yaml 
file.  
   
   My namespace for this implementation was colosseum.
   
   I had a custom yaml filed called values.yaml in the same working directory 
that I was performing my kubectl commands.
   I also had my git secret ready in the same working directory.
   
   kubectl commands I used are below:
   
   `kubectl create namespace colosseum --dry-run=client -o yaml | kubectl apply 
-f -`
   `kubectl create secret generic airflow-gke-git-secret 
--from-file=gitSshKey=airflow-gke-git-secret --namespace colosseum 
--dry-run=client -o yaml | kubectl apply -f -`
   `helm upgrade --install my-airflow apache-airflow/airflow --namespace 
colosseum --create-namespace -f values.yaml --debug`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #17977: DagRun marked as success without executing task

2023-03-02 Thread via GitHub 


github-actions[bot] closed issue #17977: DagRun marked as success without 
executing task
URL: https://github.com/apache/airflow/issues/17977


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #17977: DagRun marked as success without executing task

2023-03-02 Thread via GitHub 


github-actions[bot] commented on issue #17977:
URL: https://github.com/apache/airflow/issues/17977#issuecomment-1452821207

   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #24652: Add ``@task.snowpark`` decorator

2023-03-02 Thread via GitHub 


github-actions[bot] closed pull request #24652: Add ``@task.snowpark`` decorator
URL: https://github.com/apache/airflow/pull/24652


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] hussein-awala commented on issue #29833: Celery tasks stuck in queued state after worker crash (Set changed size during iteration)

2023-03-02 Thread via GitHub 


hussein-awala commented on issue #29833:
URL: https://github.com/apache/airflow/issues/29833#issuecomment-1452785595

   Can you extract the ti state from the DB for a task stuck in queued state?
   ```sql
   SELECT *
   FROM task_instance
   WHERE dag_id='' AND task_id='' AND 
execution_date=''
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29609: Decouple production executor warning in dags UI

2023-03-02 Thread via GitHub 


o-nikolas commented on PR #29609:
URL: https://github.com/apache/airflow/pull/29609#issuecomment-1452761063

   Thoughts on the above anyone?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29469: Updates to provider release verification documentation and check_files to produce a more useful Dockerfile

2023-03-02 Thread via GitHub 


o-nikolas commented on PR #29469:
URL: https://github.com/apache/airflow/pull/29469#issuecomment-1452742994

   @potiuk I think this one is now ready for another round of review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] hussein-awala commented on a diff in pull request #29809: Adding configuration to control retry parameters for k8s api client

2023-03-02 Thread via GitHub 


hussein-awala commented on code in PR #29809:
URL: https://github.com/apache/airflow/pull/29809#discussion_r1123866188


##
airflow/kubernetes/kube_client.py:
##
@@ -105,16 +107,24 @@ def get_kube_client(
 if conf.getboolean("kubernetes_executor", "enable_tcp_keepalive"):
 _enable_tcp_keepalive()
 
+new_client_config = Configuration.get_default_copy()
+api_client_retry_configuration = conf.getjson("kubernetes", 
"api_client_retry_configuration", fallback={})
+
+if not conf.getboolean("kubernetes_executor", "verify_ssl"):
+_disable_verify_ssl()
+
+if api_client_retry_configuration != {}:
+new_client_config.retries = 
urllib3.util.Retry(api_client_retry_configuration)

Review Comment:
   ```suggestion
   if isinstance(api_client_retry_configuration, dict) and 
api_client_retry_configuration != {}:
   new_client_config.retries = 
urllib3.util.Retry(**api_client_retry_configuration)
   else:
   raise ValueError("api_client_retry_configuration should be a 
dictionary")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk closed pull request #29888: Move help message to the google auth code
URL: https://github.com/apache/airflow/pull/29888


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk commented on PR #29888:
URL: https://github.com/apache/airflow/pull/29888#issuecomment-1452721246

   Should fix failing "canary" main build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #29888: Move help message to the google auth code

2023-03-02 Thread via GitHub 


potiuk opened a new pull request, #29888:
URL: https://github.com/apache/airflow/pull/29888

   The "google-auth" 2.16.2 just released, removed the _HELP_MESSAGE that was 
imported by google provider auth util thus failing the imports in canary builds 
of ours attempting to upgrade to newver versions of released libraries.
   
   This PR inlines the original help message into our code to make it 
independent from google-auth package version used.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] raphaelauv commented on issue #27292: Support for Python 3.11 for Google Provider (upgrading all dependencies)

2023-03-02 Thread via GitHub 


raphaelauv commented on issue #27292:
URL: https://github.com/apache/airflow/issues/27292#issuecomment-1452712966

   I din't found a working constraint for google-cloud-aiplatform for 
apache-airflow-providers-google==8.10.0 with python 3.11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #27292: Support for Python 3.11 for Google Provider (upgrading all dependencies)

2023-03-02 Thread via GitHub 


potiuk commented on issue #27292:
URL: https://github.com/apache/airflow/issues/27292#issuecomment-1452700803

   Cool. Time to try 3.11 build back then.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #29828: JSON Schema Validation errors for Params prevent DAG from importing and errors are not shown

2023-03-02 Thread via GitHub 


potiuk closed issue #29828: JSON Schema Validation errors for Params prevent 
DAG from importing and errors are not shown
URL: https://github.com/apache/airflow/issues/29828


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #29828: JSON Schema Validation errors for Params prevent DAG from importing and errors are not shown

2023-03-02 Thread via GitHub 


potiuk commented on issue #29828:
URL: https://github.com/apache/airflow/issues/29828#issuecomment-1452699881

   Fixed in #29056 then. Closing as essentially duplicate of it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] raphaelauv commented on issue #27292: Support for Python 3.11 for Google Provider (upgrading all dependencies)

2023-03-02 Thread via GitHub 


raphaelauv commented on issue #27292:
URL: https://github.com/apache/airflow/issues/27292#issuecomment-1452686038

   https://github.com/googleapis/python-translate/releases/tag/v3.10.0
   oogle-cloud-translat :heavy_check_mark: 
   
   https://github.com/googleapis/python-texttospeech/releases/tag/v2.14.0
   google-cloud-texttospeech :heavy_check_mark: 
   
   https://github.com/googleapis/python-speech/releases/tag/v2.17.0
   google-cloud-speech :heavy_check_mark: 
   
   https://github.com/googleapis/python-spanner/releases/tag/v3.27.0
   google-cloud-spanner :heavy_check_mark: 
   
   https://github.com/googleapis/python-secret-manager/releases/tag/v2.15.0
   google-cloud-secret-manager :heavy_check_mark: 
   
   https://github.com/googleapis/python-language/releases/tag/v2.8.0
   google-cloud-language :heavy_check_mark: 
   
   https://github.com/googleapis/python-videointelligence/releases/tag/v2.10.0
   google-cloud-videointelligence :heavy_check_mark: 
   
   https://github.com/googleapis/python-vision/releases/tag/v3.3.0
   google-cloud-vision :heavy_check_mark: 
   
   https://github.com/googleapis/google-api-python-client/releases/tag/v2.68.0
   google-api-python-client :heavy_check_mark: 
   
   @potiuk everything is 3.11 compatible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


vandonr-amz commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123823559


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   ok, but after taking a look at it, waiter configurations don't allow 
implementing the logic needed here.
   
   Because the API doesn't allow querying status by run ID, we have to employ 
other means. The old logic was somewhat flawed but still checked the timestamp 
to make sure we weren't picking up the previous run. The new logic relies on 
the run ID. Both of those behavior are not possible with regular waiters, so I 
wouldn't block this PR on this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix Amazon ECS Enums (#29871)

2023-03-02 Thread onikolas 
This is an automated email from the ASF dual-hosted git repository.

onikolas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 76d8aaa836 Fix Amazon ECS Enums (#29871)
76d8aaa836 is described below

commit 76d8aaa8362ba199d98680d71ccb3a800cbc4d38
Author: Andrey Anshin 
AuthorDate: Fri Mar 3 02:48:05 2023 +0400

Fix Amazon ECS Enums (#29871)

* Add missing tests

* Apply suggestions from code review

Co-authored-by: Niko Oliveira 
---
 airflow/providers/amazon/aws/hooks/ecs.py  |  23 +-
 airflow/providers/amazon/aws/sensors/ecs.py|   2 +-
 tests/providers/amazon/aws/sensors/test_ecs.py | 261 +
 .../amazon/aws/waiters/test_custom_waiters.py  |  19 +-
 4 files changed, 297 insertions(+), 8 deletions(-)

diff --git a/airflow/providers/amazon/aws/hooks/ecs.py 
b/airflow/providers/amazon/aws/hooks/ecs.py
index ccaea145dd..b0d26eb224 100644
--- a/airflow/providers/amazon/aws/hooks/ecs.py
+++ b/airflow/providers/amazon/aws/hooks/ecs.py
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):
+"""
+Enum which can be compared with regular `str` and subclasses.
+
+This class avoids multiple inheritance such as AwesomeEnum(str, Enum)
+which does not work well with templated_fields and Jinja templates.
+"""
+
+def __eq__(self, other):
+if isinstance(other, str):
+return self.value == other
+return super().__eq__(other)
+
+def __hash__(self):
+return super().__hash__()  # Need to set because we redefine __eq__
+
+
+class EcsClusterStates(_StringCompareEnum):
 """Contains the possible State values of an ECS Cluster."""
 
 ACTIVE = "ACTIVE"
@@ -65,7 +82,7 @@ class EcsClusterStates(str, Enum):
 INACTIVE = "INACTIVE"
 
 
-class EcsTaskDefinitionStates(str, Enum):
+class EcsTaskDefinitionStates(_StringCompareEnum):
 """Contains the possible State values of an ECS Task Definition."""
 
 ACTIVE = "ACTIVE"
@@ -73,7 +90,7 @@ class EcsTaskDefinitionStates(str, Enum):
 DELETE_IN_PROGRESS = "DELETE_IN_PROGRESS"
 
 
-class EcsTaskStates(str, Enum):
+class EcsTaskStates(_StringCompareEnum):
 """Contains the possible State values of an ECS Task."""
 
 PROVISIONING = "PROVISIONING"
diff --git a/airflow/providers/amazon/aws/sensors/ecs.py 
b/airflow/providers/amazon/aws/sensors/ecs.py
index c1151b8f9a..d3cfacbd41 100644
--- a/airflow/providers/amazon/aws/sensors/ecs.py
+++ b/airflow/providers/amazon/aws/sensors/ecs.py
@@ -20,8 +20,8 @@ from typing import TYPE_CHECKING, Sequence
 
 import boto3
 
-from airflow import AirflowException
 from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
 from airflow.providers.amazon.aws.hooks.ecs import (
 EcsClusterStates,
 EcsHook,
diff --git a/tests/providers/amazon/aws/sensors/test_ecs.py 
b/tests/providers/amazon/aws/sensors/test_ecs.py
new file mode 100644
index 00..66d8bc5c54
--- /dev/null
+++ b/tests/providers/amazon/aws/sensors/test_ecs.py
@@ -0,0 +1,261 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TypeVar
+from unittest import mock
+
+import boto3
+import pytest
+from slugify import slugify
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.sensors.ecs import (
+DEFAULT_CONN_ID,
+EcsBaseSensor,
+EcsClusterStates,
+EcsClusterStateSensor,
+EcsHook,
+EcsTaskDefinitionStates,
+EcsTaskDefinitionStateSensor,
+EcsTaskStates,
+EcsTaskStateSensor,
+)
+from airflow.utils import timezone
+from airflow.utils.types import NOTSET
+
+_Operator = TypeVar("_Operator")
+TEST_CLUSTER_NAME = "fake-cluster"
+TEST_TASK_ARN = "arn:aws:ecs:us-east-1:012345678910:task/spam-egg"
+TEST_TASK_DEFINITION_ARN = 
"arn:aws:ecs:us-east-1:012345678910:task-definition/foo-bar:42"
+
+
+class EcsBaseTestCase:
+@pytest.fixture(autouse=True)
+def setup_test_cases(self, monkeypatch, request,

[GitHub] [airflow] o-nikolas merged pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


o-nikolas merged PR #29871:
URL: https://github.com/apache/airflow/pull/29871


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SamWheating commented on issue #29828: JSON Schema Validation errors for Params prevent DAG from importing and errors are not shown

2023-03-02 Thread via GitHub 


SamWheating commented on issue #29828:
URL: https://github.com/apache/airflow/issues/29828#issuecomment-1452638928

   This was inadvertently fixed in 
https://github.com/apache/airflow/pull/29056, where the `Except` in 
`_process_modules` was modified to catch any exceptions. 
   
   Can confirm that these issues are correctly surfaced when running Airflow 
from `main`:
   https://user-images.githubusercontent.com/16950874/222573097-6dffd0e7-1acf-4454-95da-0eb305c88abe.png;>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] agentdanger commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


agentdanger commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452631098

   I'm caveating my "no issue" suggestion above.  I'm currently working through 
bugs on the git-sync but those could be my own doing.  My work around was able 
to solve the postgresql error I was getting...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] agentdanger commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


agentdanger commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452620409

   I was able to resolve this issue.  
   
   Upgrading also refers to upgrading the existing 1.8 instance.  You have to 
follow @potiuk suggestion within the release notes.
   
   `kubectl delete secret {RELEASE_NAME}-postgresql`
   `kubectl delete statefulset {RELEASE_NAME}-postgresql`
   
   Then in your values.yaml file.  You must set the username and password in 
the postgresql subchart configuration:
   
   postgresql:
 enabled: true
 image:
   tag: "11"
 auth:
   enablePostgresUser: true
   postgresPassword: postgres
   username: **YOUR_USERNAME**
   password: **YOUR_PASSWORD**
   
   No issues once I did all of those steps.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SamWheating commented on issue #29844: ComputeEngineInsertInstanceFromTemplateOperator fails to parse SEV_SNP_CAPABLE in guestOsFeatures disk field

2023-03-02 Thread via GitHub 


SamWheating commented on issue #29844:
URL: https://github.com/apache/airflow/issues/29844#issuecomment-1452592062

   Could you run `pip list` and share your results? I suspect that this is due 
to an older version of the `google-cloud-compute` library. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] agentdanger commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


agentdanger commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452576191

   I get the issue when I do a helm upgrade to install the custom values.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] agentdanger commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


agentdanger commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452573578

   I can recreate this issue by doing a fresh install using the 1.8 helm image 
"my-airflow", in a fresh namespace on kubernetes, and customizing the 
values.yaml to attempt to turn on git-sync.  I have the same issue the original 
OP is mentioning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-03-02 Thread via GitHub 


ferruzzi commented on issue #29640:
URL: https://github.com/apache/airflow/issues/29640#issuecomment-1452530133

   @antonio-antuan  I am trying to reopen [the 
ticket](https://github.com/boto/botocore/issues/2608) that Taragolis found on 
the botocore side.  It looks like it has been raised a couple times but gets 
dropped.  Maybe we can help them sort it out.  Would you mind keeping an eye on 
their ticket in case they ask for more details I don't have?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (372ea492e5 -> ab066b5823)

2023-03-02 Thread jedcunningham 
This is an automated email from the ASF dual-hosted git repository.

jedcunningham pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 372ea492e5 Update sftp_to_s3.rst (#29869)
 add ab066b5823 Add note about k8s version support (#29884)

No new revisions were added by this update.

Summary of changes:
 chart/README.md   | 2 +-
 dev/breeze/src/airflow_breeze/global_constants.py | 5 +
 docs/helm-chart/index.rst | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)

[GitHub] [airflow] jedcunningham merged pull request #29884: Add note about k8s version support

2023-03-02 Thread via GitHub 


jedcunningham merged PR #29884:
URL: https://github.com/apache/airflow/pull/29884


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29772: Support deleting the local log files when using remote logging

2023-03-02 Thread via GitHub 


o-nikolas commented on PR #29772:
URL: https://github.com/apache/airflow/pull/29772#issuecomment-1452510296

   Looks like there are still some failing tests @hussein-awala 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas merged pull request #29869: Update sftp_to_s3.rst

2023-03-02 Thread via GitHub 


o-nikolas merged PR #29869:
URL: https://github.com/apache/airflow/pull/29869


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Update sftp_to_s3.rst (#29869)

2023-03-02 Thread onikolas 
This is an automated email from the ASF dual-hosted git repository.

onikolas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 372ea492e5 Update sftp_to_s3.rst (#29869)
372ea492e5 is described below

commit 372ea492e5180ff70320bc5d6789517c087356ac
Author: Michał Wesołowski <12141793+wesolows...@users.noreply.github.com>
AuthorDate: Thu Mar 2 21:31:16 2023 +0100

Update sftp_to_s3.rst (#29869)

This operator does not use AWS transfer family. it just connects to SFTP, 
copies the file locally and puts it on S3.
---
 docs/apache-airflow-providers-amazon/operators/transfer/sftp_to_s3.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/docs/apache-airflow-providers-amazon/operators/transfer/sftp_to_s3.rst 
b/docs/apache-airflow-providers-amazon/operators/transfer/sftp_to_s3.rst
index 67f91c9edb..3023d7bf7e 100644
--- a/docs/apache-airflow-providers-amazon/operators/transfer/sftp_to_s3.rst
+++ b/docs/apache-airflow-providers-amazon/operators/transfer/sftp_to_s3.rst
@@ -20,7 +20,7 @@ SFTP to Amazon S3
 =
 
 Use the ``SFTPToS3Operator`` transfer to copy the data from a SFTP server to 
an Amazon Simple Storage Service (S3) file.
-For more information about the service visits `Amazon Transfer for SFTP API 
documentation 
`_.
+
 
 Prerequisite Tasks
 --

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29869: Update sftp_to_s3.rst

2023-03-02 Thread via GitHub 


boring-cyborg[bot] commented on PR #29869:
URL: https://github.com/apache/airflow/pull/29869#issuecomment-1452505556

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (9745032fa7 -> faf792f634)

2023-03-02 Thread onikolas 
This is an automated email from the ASF dual-hosted git repository.

onikolas pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 9745032fa7 including airflow/example_dags/sql/sample.sql in 
MANIFEST.in (#29883)
 add faf792f634 Implement custom boto waiters for some EMR operators 
(#29822)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/amazon/aws/operators/emr.py  | 173 +++--
 airflow/providers/amazon/aws/utils/waiter.py   |   4 +
 airflow/providers/amazon/aws/waiters/emr.json  |  80 ++
 tests/providers/amazon/aws/hooks/test_emr.py   |   7 +
 tests/providers/amazon/aws/operators/test_eks.py   |  12 +-
 .../aws/operators/test_emr_create_job_flow.py  |  24 ++-
 .../aws/operators/test_emr_notebook_execution.py   | 117 --
 tests/providers/amazon/aws/utils/test_waiter.py|  11 ++
 8 files changed, 315 insertions(+), 113 deletions(-)
 create mode 100644 airflow/providers/amazon/aws/waiters/emr.json

[GitHub] [airflow] o-nikolas merged pull request #29822: Implement custom boto waiters for some EMR operators

2023-03-02 Thread via GitHub 


o-nikolas merged PR #29822:
URL: https://github.com/apache/airflow/pull/29822


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SamWheating opened a new pull request, #29886: Mask out non-access bits when comparing file modes

2023-03-02 Thread via GitHub 


SamWheating opened a new pull request, #29886:
URL: https://github.com/apache/airflow/pull/29886

   While investigating https://github.com/apache/airflow/issues/29112 I found 
another issue in the way that the file task handler checks file access:
   
   
https://github.com/apache/airflow/blob/9745032fa7e05ea7e45266445db44d153c1adb11/airflow/utils/log/file_task_handler.py#L439-L445
   
   The return value of `path.stat().st_mode` will include additional bits 
beyond the 9 access bits (`-RWXRWSRWX`)
   
   As demonstrated here:
   ```python
   >>> from pathlib import Path
   
   >>> p = Path('/opt/airflow/logs/task/mytask')
   
   >>> new_folder_permissions = 0o755
   
   >>> p.mkdir(mode=new_folder_permissions, parents=True, exist_ok=True)
   
   >>> p.stat().st_mode == new_folder_permissions   # Should return true, 
because we just created this directory
   False
   
   >>> format(p.stat().st_mode, 'o')   # Lets look at the full value of 
`std_mode`, oct-formatted
   '42755'
   
   >>> p.stat().st_mode % 0o1000 == new_folder_permissions   # masking out the 
last 9 bits
   True
   ```
   
   While I don't think that this is the only cause of 
https://github.com/apache/airflow/issues/29112, it might lead to unnecessarily 
chmod'ing directories which could then throw errors. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


potiuk commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452466045

   Yes. This is a known "expectation" for chart 1.8.0 and the way how to deal 
with it is described in [release 
notes](https://airflow.apache.org/docs/helm-chart/stable/release_notes.html#airflow-helm-chart-1-8-0-2023-02-06)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] PierreC1024 commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


PierreC1024 commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123625999


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   I try to do like this but currently it is not possible to define 
`project_id` as an attribute. 
   It seems this is because of `GoogleBaseHook` have defined `project_id` with 
a decorated @property and it does not have a setter method for `project_id` 
([see](https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py#L342)).
  
   
   I am not confortable to change it in this PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] PierreC1024 commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


PierreC1024 commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123625999


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   I try to do like this but currently it is not possible to define 
`project_id` as an attribute. 
   This is because of `GoogleBaseHook` have defined `project_id` with a 
decorated @property and it does not have a setter method for `project_id` 
([see](https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py#L342)).
  
   
   I am not confortable to change it in this PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #18329: Add task name, DAG name, try_number, and run_id to all Kubernetes executor logs

2023-03-02 Thread via GitHub 


potiuk commented on issue #18329:
URL: https://github.com/apache/airflow/issues/18329#issuecomment-1452458530

   Feel free


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (b33b11bf9f -> 9745032fa7)

2023-03-02 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from b33b11bf9f Update ref anchor for env var link in Connection how-to doc 
(#29816)
 add 9745032fa7 including airflow/example_dags/sql/sample.sql in 
MANIFEST.in (#29883)

No new revisions were added by this update.

Summary of changes:
 MANIFEST.in | 1 +
 1 file changed, 1 insertion(+)

[GitHub] [airflow] potiuk merged pull request #29883: Including airflow/example_dags/sql/sample.sql in MANIFEST.in

2023-03-02 Thread via GitHub 


potiuk merged PR #29883:
URL: https://github.com/apache/airflow/pull/29883


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SamWheating commented on issue #29112: "Operation not permitted" error when chmod on log folder

2023-03-02 Thread via GitHub 


SamWheating commented on issue #29112:
URL: https://github.com/apache/airflow/issues/29112#issuecomment-1452445056

   I have been trying to replicate this issue this morning, here's what I did:
   
- set up a Kind cluster with a ReadWriteMany PVC (running on NFS)
- Deployed Airflow 2.5.1 with the latest helm chart, writing task logs to 
NFS
- Ran some tasks, which all succeeded.

So I think that it must be an issue of the access / fsGroup on your volume, 
and might be related to the CSI? 

If anyone has any other suggestions, I am happy to work on a fix, but would 
like to replicate the issue first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi opened a new pull request, #29885: Add filter task upstream/downstream to grid view

2023-03-02 Thread via GitHub 


bbovenzi opened a new pull request, #29885:
URL: https://github.com/apache/airflow/pull/29885

   Add new filtering options in #29226 to the grid view filters and apply it to 
both grid_data and graph_data endpoints.
   
   Also, allow filtering tasks and resetting the root without a full reload of 
the grid data page.
   
   Related: #29852
   
   ![Mar-02-2023 
14-20-58](https://user-images.githubusercontent.com/4600967/222530261-c7f43ed9-d494-46ef-a896-ac694f688c68.gif)
   
   https://user-images.githubusercontent.com/4600967/222530326-ab242a62-ffdf-4aeb-b195-f3aae0de0704.png;>
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vincbeck commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


vincbeck commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123591998


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   Sounds good to me :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] empire-sports commented on issue #29444: Attempting to Upgrade Chart to Enable Examples Causes Postgres Error

2023-03-02 Thread via GitHub 


empire-sports commented on issue #29444:
URL: https://github.com/apache/airflow/issues/29444#issuecomment-1452416237

   Commenting on this.  I am upgrading from 1.7 to 1.8.  I used a custom 
values.yaml file with my implementation to enable the git-sync.  I'm receiving 
a similar error.
   
   `namespace/colosseum configured
   secret/airflow-gke-git-secret configured
   history.go:56: [debug] getting history for release airflow
   upgrade.go:144: [debug] preparing upgrade for airflow
   Error: UPGRADE FAILED: execution error at 
(airflow/charts/postgresql/templates/secrets.yaml:17:24): 
   PASSWORDS ERROR: The secret "airflow-postgresql" does not contain the key 
"postgres-password"
   
   helm.go:84: [debug] execution error at 
(airflow/charts/postgresql/templates/secrets.yaml:17:24): 
   PASSWORDS ERROR: The secret "airflow-postgresql" does not contain the key 
"postgres-password"`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


Taragolis commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123587735


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   I'd rather move it in separate PR when we actually need this functional. 
Just for avoid spending time to choose better name and better place 藍  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vincbeck commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


vincbeck commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123582113


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   Agree. We might want to do it now? It'd encourage others to use if needed 
instead of copy pasting it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29878: Prepare docs for 03/2023 wave of Providers

2023-03-02 Thread via GitHub 


jedcunningham commented on code in PR #29878:
URL: https://github.com/apache/airflow/pull/29878#discussion_r1123583646


##
airflow/providers/mysql/CHANGELOG.rst:
##
@@ -27,6 +27,14 @@ used with MySQL server 5.6.4 through 5.7.
 Changelog
 -
 
+4.0.2
+.
+
+Misc
+
+
+* ``Use MariaDB client binaries in arm64 image for support MySQL backend 
(#29519)``

Review Comment:
   Yep, for sure. I was just thinking an easy interim solution.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


o-nikolas commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123582402


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   I feel like it is quite trivial to move it to a more generic place (and 
still keep it private if you'd prefer that). Seems easy to make the right 
choice now rather than having to fix it later. But I wont press any harder, 
happy to merge this as is and I can fix it in a follow up PR :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham closed pull request #29882: Drop support for k8s 1.23

2023-03-02 Thread via GitHub 


jedcunningham closed pull request #29882: Drop support for k8s 1.23
URL: https://github.com/apache/airflow/pull/29882


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


ferruzzi commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123575711


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   I definitely agree that this should be a waiter and may as well be done now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


ferruzzi commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123575711


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   I definitely agree that this should be a waiter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on pull request #29882: Drop support for k8s 1.23

2023-03-02 Thread via GitHub 


jedcunningham commented on PR #29882:
URL: https://github.com/apache/airflow/pull/29882#issuecomment-1452395447

   Okay, opened #29884 so it's more visible!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham opened a new pull request, #29884: Add note about k8s version support

2023-03-02 Thread via GitHub 


jedcunningham opened a new pull request, #29884:
URL: https://github.com/apache/airflow/pull/29884

   This will help prevent us accidentally dropping support for k8s versions 
earlier than we've agreed to it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #29878: Prepare docs for 03/2023 wave of Providers

2023-03-02 Thread via GitHub 


eladkal commented on code in PR #29878:
URL: https://github.com/apache/airflow/pull/29878#discussion_r1123572510


##
airflow/providers/mysql/CHANGELOG.rst:
##
@@ -27,6 +27,14 @@ used with MySQL server 5.6.4 through 5.7.
 Changelog
 -
 
+4.0.2
+.
+
+Misc
+
+
+* ``Use MariaDB client binaries in arm64 image for support MySQL backend 
(#29519)``

Review Comment:
   We know we want to move to newsfragment... just no one had the time to make 
it happen yet



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29869: Update sftp_to_s3.rst

2023-03-02 Thread via GitHub 


o-nikolas commented on PR #29869:
URL: https://github.com/apache/airflow/pull/29869#issuecomment-1452382584

   > docs link in the PR description is old (1.10.3). The new docs is 
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/transfers/sftp_to_s3.html#SFTPToS3Operator.
 Do you still think this change require?
   
   That link is old, but the code still basically functions as described in the 
old link:
   
https://github.com/apache/airflow/blob/eb3a96dfa6ab17360eb43b901f05d082dc318971/airflow/providers/amazon/aws/transfers/sftp_to_s3.py#L82-L96


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29882: Drop support for k8s 1.23

2023-03-02 Thread via GitHub 


potiuk commented on PR #29882:
URL: https://github.com/apache/airflow/pull/29882#issuecomment-1452381931

   > Hmm, I wasn't aware of the k8s exception. Let me have a think on that. 
Fwiw, as of today 2 support 1.22 still.
   
   Yep. I noticed :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi closed issue #23169: Task Duration shows as "empty" when it was really less than 1 second

2023-03-02 Thread via GitHub 


bbovenzi closed issue #23169: Task Duration shows as "empty" when it was really 
less than 1 second
URL: https://github.com/apache/airflow/issues/23169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #23169: Task Duration shows as "empty" when it was really less than 1 second

2023-03-02 Thread via GitHub 


bbovenzi commented on issue #23169:
URL: https://github.com/apache/airflow/issues/23169#issuecomment-1452381187

   https://user-images.githubusercontent.com/4600967/222523140-4728de05-80fa-40ca-b61b-aa2e9f54e459.png;>
   
   We got rid of the tree view a long time ago and we know calculate duration 
better.
   
   Even the legacy tooltip in the graph seems to be working:
   https://user-images.githubusercontent.com/4600967/222523296-2696fbd4-94c5-4f36-a7a3-a43544be94f5.png;>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29878: Prepare docs for 03/2023 wave of Providers

2023-03-02 Thread via GitHub 


jedcunningham commented on code in PR #29878:
URL: https://github.com/apache/airflow/pull/29878#discussion_r1123558759


##
airflow/providers/mysql/CHANGELOG.rst:
##
@@ -27,6 +27,14 @@ used with MySQL server 5.6.4 through 5.7.
 Changelog
 -
 
+4.0.2
+.
+
+Misc
+
+
+* ``Use MariaDB client binaries in arm64 image for support MySQL backend 
(#29519)``

Review Comment:
   Oh, well that's a serious bummer. Commit messages aren't necessarily good 
changelog entries. We should look at loosening that to look for the PR number 
instead to correlate.
   
   I don't think it's worth doing anything extra for this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #23169: Task Duration shows as "empty" when it was really less than 1 second

2023-03-02 Thread via GitHub 


amoghrajesh commented on issue #23169:
URL: https://github.com/apache/airflow/issues/23169#issuecomment-1452354467

   @eladkal @bbovenzi I can take this issue if nobody is working on it. Seems 
like an interesting one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pankajastro commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


pankajastro commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123551136


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   hmm, in that case I would prefer to have this in 
`BigQueryReservationServiceHook`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #18329: Add task name, DAG name, try_number, and run_id to all Kubernetes executor logs

2023-03-02 Thread via GitHub 


amoghrajesh commented on issue #18329:
URL: https://github.com/apache/airflow/issues/18329#issuecomment-1452345107

   @potiuk seems like the solution was not completed here. I would like to try 
my hand at it if it is available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] PierreC1024 commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


PierreC1024 commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123543321


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   Sometimes, you wish to create bi_engine reservation not on the same project 
than the default project (defining in the connection). 
   I choose to put it as the function argument but we could add it in the 
`BigQueryReservationServiceHook` attribute. What do you prefer ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] PierreC1024 commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


PierreC1024 commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123543321


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   Most of the time you wish to create bi_engine reservation not on the same 
project than the default project (defining in the connection). 
   I choose to put it as the function argument but we could add it in the 
`BigQueryReservationServiceHook` attribute. What do you prefer ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SamWheating opened a new pull request, #29883: Including airflow/example_dags/sql/sample.sql in MANIFEST.in

2023-03-02 Thread via GitHub 


SamWheating opened a new pull request, #29883:
URL: https://github.com/apache/airflow/pull/29883

   I was debugging another issue and running the official helm chart in Kind, 
and when I tried to run the `example_python_operator` DAG it failed on the 
`log_sql_query` task with the following error:
   
   ```
 File 
"/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 
969, in _load_template
   template = self.loader.load(self, name, self.make_globals(globals))
 File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/loaders.py", 
line 126, in load
   source, filename, uptodate = self.get_source(environment, name)
 File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/loaders.py", 
line 218, in get_source
   raise TemplateNotFound(template)
   jinja2.exceptions.TemplateNotFound: sql/sample.sql
   ```
   
   Which then causes all of the downstream tasks to fail.
   
   This is because the template used in the example DAG isn't bundled into the 
build. I suspect this might work in `breeze` because the source directory is 
mounted, but when Airflow is installed from pip it won't include this file. 
   
   Anyways, I think that this minor change should fix things.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pankajastro commented on a diff in pull request #29835: Provider Google Cloud - Add bigquery bi engine reservation operators

2023-03-02 Thread via GitHub 


pankajastro commented on code in PR #29835:
URL: https://github.com/apache/airflow/pull/29835#discussion_r1123530878


##
airflow/providers/google/cloud/hooks/bigquery_reservation.py:
##
@@ -0,0 +1,125 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains a BigQuery Reservation Hook."""
+from __future__ import annotations
+
+from typing import Sequence
+
+from google.cloud.bigquery_reservation_v1 import ReservationServiceClient
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.consts import CLIENT_INFO
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class BigQueryReservationServiceHook(GoogleBaseHook):
+"""
+Hook for Google Bigquery Reservation API.
+
+:param gcp_conn_id: The Airflow connection used for GCP credentials.
+:param location: The location of the BigQuery resource.
+:param impersonation_chain: This is the optional service account to 
impersonate using short term
+credentials.
+"""
+
+conn_name_attr = "gcp_conn_id"
+default_conn_name = "google_cloud_bigquery_reservation_default"
+hook_name = "Google Bigquery Reservation"
+
+def __init__(
+self,
+gcp_conn_id: str = GoogleBaseHook.default_conn_name,
+location: str | None = None,
+impersonation_chain: str | Sequence[str] | None = None,
+) -> None:
+super().__init__(
+gcp_conn_id=gcp_conn_id,
+impersonation_chain=impersonation_chain,
+)
+self.location = location
+self._client: ReservationServiceClient | None = None
+
+@staticmethod
+def _convert_gb_to_kb(value: int) -> int:
+"""
+Convert GB value to KB.
+
+:param value: Value to convert
+"""
+return value * 1073741824
+
+def get_client(self) -> ReservationServiceClient:
+"""
+Get reservation service client.
+
+:return: Google Bigquery Reservation client
+"""
+if not self._client:
+self._client = ReservationServiceClient(
+credentials=self.get_credentials(), client_info=CLIENT_INFO
+)
+return self._client
+
+@GoogleBaseHook.fallback_to_default_project_id
+def create_bi_reservation(self, project_id: str, size: int) -> None:

Review Comment:
   do we need project_id args here? can we access this with `self.project_id` 
since this is a base class property? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


ferruzzi commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123531685


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   I was thinking the same thing.  Seems like something we'll want to reuse.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


Taragolis commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123531262


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   Right now it just for fix but in general we could move it later if it useful 
for other places, that why I make it "private"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


ferruzzi commented on PR #29871:
URL: https://github.com/apache/airflow/pull/29871#issuecomment-1452321027

   Thanks for getting this so fast, I was planning to look at it this morning.  
:+1: 
   
   LOL.  I did a review and it wouldn't let me submit because Niko left the 
same two comments while I was typing.  :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vincbeck commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


vincbeck commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123525633


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   To me it should be done in this PR. It is basically the same problem with 2 
different implementations, let's aim directly the desire implementation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vincbeck commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


vincbeck commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123525633


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   To me it should be done in this PR. It is basically the same problem with 2 
different implementation, let's aim directly the desire implementation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29871: Fix Amazon ECS Enums

2023-03-02 Thread via GitHub 


o-nikolas commented on code in PR #29871:
URL: https://github.com/apache/airflow/pull/29871#discussion_r1123521462


##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):

Review Comment:
   Should we put this in a more general place so that other modules can make 
use of it? 



##
airflow/providers/amazon/aws/hooks/ecs.py:
##
@@ -55,7 +55,24 @@ def should_retry_eni(exception: Exception):
 return False
 
 
-class EcsClusterStates(str, Enum):
+class _StringCompareEnum(Enum):
+"""
+Enum which could be compared with regular `str` and subclasses.
+
+This class for avoid multiple inheritance such as AwesomeEnum(str, Enum)
+which is not work well with templated_fields, and Jinja templates.

Review Comment:
   ```suggestion
   Enum which can be compared with regular `str` and subclasses.
   
   This class avoids multiple inheritance such as AwesomeEnum(str, Enum)
   which does not work well with templated_fields and Jinja templates.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell merged pull request #29816: Fix ref anchor for env var link in Connection how-to doc

2023-03-02 Thread via GitHub 


josh-fell merged PR #29816:
URL: https://github.com/apache/airflow/pull/29816


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Update ref anchor for env var link in Connection how-to doc (#29816)

2023-03-02 Thread joshfell 
This is an automated email from the ASF dual-hosted git repository.

joshfell pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new b33b11bf9f Update ref anchor for env var link in Connection how-to doc 
(#29816)
b33b11bf9f is described below

commit b33b11bf9fdbbedbd31699428853896f4db478cb
Author: Josh Fell <48934154+josh-f...@users.noreply.github.com>
AuthorDate: Thu Mar 2 13:05:18 2023 -0500

Update ref anchor for env var link in Connection how-to doc (#29816)

In the "Managing Connections" how-to doc, the "environment variables" links 
in the doc point to the "Security of connections in the database" section 
rather than the "Storing connections in environment variables" section.
---
 docs/apache-airflow-providers-dbt-cloud/connections.rst | 2 +-
 docs/apache-airflow/cli-and-env-variables-ref.rst   | 2 +-
 docs/apache-airflow/howto/connection.rst| 7 ---
 3 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/docs/apache-airflow-providers-dbt-cloud/connections.rst 
b/docs/apache-airflow-providers-dbt-cloud/connections.rst
index 2b0509dbab..1bd3945577 100644
--- a/docs/apache-airflow-providers-dbt-cloud/connections.rst
+++ b/docs/apache-airflow-providers-dbt-cloud/connections.rst
@@ -102,5 +102,5 @@ For example, to add a connection with the connection ID of 
"dbt_cloud_default":
 export 
AIRFLOW_CONN_DBT_CLOUD_DEFAULT='dbt-cloud://:api_to...@my-tenant.getdbt.com'
 
 You can refer to the documentation on
-:ref:`creating connections via environment variables 
` for more
+:ref:`creating connections via environment variables 
` for more
 information.
diff --git a/docs/apache-airflow/cli-and-env-variables-ref.rst 
b/docs/apache-airflow/cli-and-env-variables-ref.rst
index 7593233585..e2a929f1f4 100644
--- a/docs/apache-airflow/cli-and-env-variables-ref.rst
+++ b/docs/apache-airflow/cli-and-env-variables-ref.rst
@@ -90,7 +90,7 @@ Environment Variables
   For example, if you want to create a connection named 
``PROXY_POSTGRES_TCP``, you can create
   a key ``AIRFLOW_CONN_PROXY_POSTGRES_TCP`` with the connection URI as the 
value.
 
-  For more information, see: :ref:`environment_variables_secrets_backend`.
+  For more information, see: :ref:`environment_variables_connections`.
 
 .. envvar:: AIRFLOW_HOME
 
diff --git a/docs/apache-airflow/howto/connection.rst 
b/docs/apache-airflow/howto/connection.rst
index 9b0e127f14..00b99de8ae 100644
--- a/docs/apache-airflow/howto/connection.rst
+++ b/docs/apache-airflow/howto/connection.rst
@@ -26,11 +26,13 @@ Airflow's :class:`~airflow.models.connection.Connection` 
object is used for stor
 
 Connections may be defined in the following ways:
 
-  - in :ref:`environment variables `
+  - in :ref:`environment variables `
   - in an external 
:doc:`/administration-and-deployment/security/secrets/secrets-backend/index`
   - in the :ref:`Airflow metadata database `
 (using the :ref:`CLI ` or :ref:`web UI 
`)
 
+.. _environment_variables_connections:
+
 Storing connections in environment variables
 
 
@@ -92,7 +94,7 @@ Storing connections in the database
 ---
 .. seealso::
 
-Connections can alternatively be stored in :ref:`environment variables 
` or an :doc:`external secrets backend 
` such 
as HashiCorp Vault, AWS SSM Parameter Store, etc.
+Connections can alternatively be stored in :ref:`environment variables 
` or an :doc:`external secrets backend 
` such 
as HashiCorp Vault, AWS SSM Parameter Store, etc.
 
 When storing connections in the database, you may manage them using either the 
web UI or the Airflow CLI.
 
@@ -179,7 +181,6 @@ Exporting connections to file
 
 You can export to file connections stored in the database (e.g. for migrating 
connections from one environment to another).  See :ref:`Exporting Connections 
` for usage.
 
-.. _environment_variables_secrets_backend:
 
 Security of connections in the database
 ^^^

[GitHub] [airflow] josh-fell commented on pull request #29816: Fix ref anchor for env var link in Connection how-to doc

2023-03-02 Thread via GitHub 


josh-fell commented on PR #29816:
URL: https://github.com/apache/airflow/pull/29816#issuecomment-1452306725

   > We can merge but I guess since releases are not synced there will be some 
time that the reference won't work
   
   I think that's fine. It's the only provider that uses this link and there 
are a few examples of creating the connection with env vars too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-03-02 Thread via GitHub 


vandonr-amz commented on code in PR #28869:
URL: https://github.com/apache/airflow/pull/28869#discussion_r1123512646


##
airflow/providers/amazon/aws/hooks/appflow.py:
##
@@ -54,49 +50,35 @@ def conn(self) -> AppflowClient:
 """Get the underlying boto3 Appflow client (cached)"""
 return super().conn
 
-def run_flow(self, flow_name: str, poll_interval: int = 20) -> str:
+def run_flow(self, flow_name: str, poll_interval: int = 20, 
wait_for_completion: bool = True) -> str:
 """
 Execute an AppFlow run.
 
 :param flow_name: The flow name
 :param poll_interval: Time (seconds) to wait between two consecutive 
calls to check the run status
+:param wait_for_completion: whether to wait for the run to end to 
return
 :return: The run execution ID
 """
-ts_before: datetime = datetime.now(timezone.utc)
-sleep(self.EVENTUAL_CONSISTENCY_OFFSET)
 response_start = self.conn.start_flow(flowName=flow_name)
 execution_id = response_start["executionId"]
 self.log.info("executionId: %s", execution_id)
 
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait Appflow eventual consistence
-self.log.info("Waiting for Appflow eventual consistence...")
-while (
-response_desc.get("lastRunExecutionDetails", {}).get(
-"mostRecentExecutionTime", datetime(1970, 1, 1, 
tzinfo=timezone.utc)
-)
-< ts_before
-):
-sleep(self.EVENTUAL_CONSISTENCY_POLLING)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-# Wait flow stops
-self.log.info("Waiting for flow run...")
-while (
-"mostRecentExecutionStatus" not in last_exec_details
-or last_exec_details["mostRecentExecutionStatus"] == "InProgress"
-):
-sleep(poll_interval)
-response_desc = self.conn.describe_flow(flowName=flow_name)
-last_exec_details = response_desc["lastRunExecutionDetails"]
-
-self.log.info("lastRunExecutionDetails: %s", last_exec_details)
-
-if last_exec_details["mostRecentExecutionStatus"] == "Error":
-raise Exception(f"Flow error:\n{json.dumps(response_desc, 
default=str)}")
+if wait_for_completion:

Review Comment:
   ok, but do you think it should be done as part of this PR ? How about I push 
the conversion to waiter as a new PR ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] shyft-mike commented on a diff in pull request #28846: Updated app to support configuring the caching hash method for FIPS

2023-03-02 Thread via GitHub 


shyft-mike commented on code in PR #28846:
URL: https://github.com/apache/airflow/pull/28846#discussion_r1123510383


##
airflow/compat/hashlib_wrapper.py:
##
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import hashlib
+import sys
+from typing import Any
+
+# Check if "usedforsecurity" is available for hashlib
+if sys.version_info >= (3, 9):
+HAS_USEDFORSECURITY = True
+else:
+HAS_USEDFORSECURITY = False
+
+
+def md5(data: Any, used_for_security: bool | None = None):

Review Comment:
   Sorry, I thought I responded. I moved things under `utils`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on pull request #29882: Drop support for k8s 1.23

2023-03-02 Thread via GitHub 


jedcunningham commented on PR #29882:
URL: https://github.com/apache/airflow/pull/29882#issuecomment-1452270108

   Hmm, I wasn't aware of the k8s exception. Let me have a think on that. Fwiw, 
as of today 2 support 1.22 still.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29882: Drop support for k8s 1.23

2023-03-02 Thread via GitHub 


potiuk commented on PR #29882:
URL: https://github.com/apache/airflow/pull/29882#issuecomment-1452269709

   (And yes, I would be happy to drop support immediately after they reach EOL, 
but it's not what we agreed to, unfortunately).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

  1   2   >