[GitHub] [airflow] eladkal opened a new pull request, #33399: revert change order of issues for stale bot

2023-08-15 Thread via GitHub
eladkal opened a new pull request, #33399: URL: https://github.com/apache/airflow/pull/33399 Revert https://github.com/apache/airflow/pull/33208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

2023-08-15 Thread via GitHub
potiuk commented on PR #33309: URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678487753 The main reason was the signature of the render_templelate method that could not be changed for the regular non-mapped operator. At least that"s what the comment says in the

[airflow] branch constraints-2-7 updated: Updating constraints. Github run id:5864038254

2023-08-15 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-2-7 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-2-7 by this push: new 3295247e67 Updating

[GitHub] [airflow] mai-nakagawa opened a new issue, #33400: BigQuery with impersonation_chain does not accept custom scopes

2023-08-15 Thread via GitHub
mai-nakagawa opened a new issue, #33400: URL: https://github.com/apache/airflow/issues/33400 ### Apache Airflow version main (development) ### What happened I always face the following error when I try to run a BigQuery query that accesses [connected

[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

2023-08-15 Thread via GitHub
potiuk commented on PR #33309: URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678489645 Or maybe you think we can change the signature ? I believe (on the phone now) the original comment mentioned that the signature cannot be changed because the methods that would have to

[GitHub] [airflow] eladkal merged pull request #33399: revert change order of issues for stale bot

2023-08-15 Thread via GitHub
eladkal merged PR #33399: URL: https://github.com/apache/airflow/pull/33399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: revert change order of issues for stale bot (#33399)

2023-08-15 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9927bdaa88 revert change order of issues for

[GitHub] [airflow] Lee-W opened a new pull request, #33401: Respect soft fail argument when poke is called

2023-08-15 Thread via GitHub
Lee-W opened a new pull request, #33401: URL: https://github.com/apache/airflow/pull/33401 This is the following PR to https://github.com/apache/airflow/pull/33196 and intends to solve the issue that the `soft_fail` argument might not work on all sensors. This PR handles the case

[GitHub] [airflow] arjunanan6 opened a new issue, #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
arjunanan6 opened a new issue, #33402: URL: https://github.com/apache/airflow/issues/33402 ### Apache Airflow version 2.6.3 ### What happened This is a new problem that started happening overnight for us on only one of our environments. The only change introduced was to

[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

2023-08-15 Thread via GitHub
potiuk commented on PR #33309: URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678486007 The main reason was the signature of the render_templelate method that could not be changed for the regular non-mapped operator. At least that"s what the comment says in the

[GitHub] [airflow] RNHTTR commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
RNHTTR commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678954965 What's the value for [delete_worker_pods](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-worker-pods)? -- This is an automated message from

[GitHub] [airflow] RNHTTR commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
RNHTTR commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678960320 Can you provide reproduction steps? I haven't seen this elsewhere in 2.6.3, and given that your environments are nearly-but-not-identical makes this sound like a configuration

[GitHub] [airflow] jedcunningham commented on a diff in pull request #33390: Sync v2-7-stable with v2-7-test to release 2.7.0

2023-08-15 Thread via GitHub
jedcunningham commented on code in PR #33390: URL: https://github.com/apache/airflow/pull/33390#discussion_r1294648953 ## RELEASE_NOTES.rst: ## @@ -37,6 +37,13 @@ Old Graph View is removed (#32958) "" The old Graph View is removed. The new

[airflow] branch main updated: Move methods from security managers to `FabAirflowSecurityManagerOverride` (#33044)

2023-08-15 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8775f841dd Move methods from security managers

[GitHub] [airflow] vincbeck merged pull request #33044: Move methods from security managers to `FabAirflowSecurityManagerOverride`

2023-08-15 Thread via GitHub
vincbeck merged PR #33044: URL: https://github.com/apache/airflow/pull/33044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: Doc: Add concrete examples for accessing context variables from TaskFlow tasks (#33296)

2023-08-15 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e57d0c913c Doc: Add concrete examples for

[airflow] branch openlineage_dont_run_tis_executor updated (9cf049c471 -> 975e1a61af)

2023-08-15 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch openlineage_dont_run_tis_executor in repository https://gitbox.apache.org/repos/asf/airflow.git omit 9cf049c471 Merge branch 'main' into openlineage_dont_run_tis_executor omit

[airflow] 01/01: openlineage: don't run task instance listener in executor

2023-08-15 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch openlineage_dont_run_tis_executor in repository https://gitbox.apache.org/repos/asf/airflow.git commit 975e1a61afa86a3c5edf56b2e5b7024dba160cfd Author: Maciej Obuchowski AuthorDate: Sun Aug

[GitHub] [airflow] vincbeck commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294686733 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1523,3 +1523,259 @@ def execute(self, context: Context) -> str: arn = ans["ExperimentArn"]

[airflow] branch main updated: Fix outdated test name and description in BatchSensor (#33407)

2023-08-15 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a1576d39d2 Fix outdated test name and

[GitHub] [airflow] Lee-W opened a new pull request, #33407: Fix outdated test name and description in BatchSensor

2023-08-15 Thread via GitHub
Lee-W opened a new pull request, #33407: URL: https://github.com/apache/airflow/pull/33407 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] arjunanan6 commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
arjunanan6 commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678787624 This new set of logs shows where the Deletion process suddenly stops: ``` [2023-08-15T11:01:58.842+] {kubernetes_executor.py:764} INFO - Deleted pod:

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294510954 ## airflow/providers/databricks/operators/databricks.py: ## @@ -160,6 +162,142 @@ def get_link( return XCom.get_value(key=XCOM_RUN_PAGE_URL_KEY,

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294511481 ## docs/apache-airflow-providers-databricks/operators/jobs_create.rst: ## @@ -0,0 +1,91 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294514250 ## airflow/providers/databricks/hooks/databricks.py: ## @@ -135,9 +137,27 @@ def __init__( ) -> None: super().__init__(databricks_conn_id,

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294521512 ## airflow/providers/databricks/hooks/databricks.py: ## @@ -135,9 +137,27 @@ def __init__( ) -> None: super().__init__(databricks_conn_id,

[GitHub] [airflow] ellisms commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
ellisms commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294549222 ## airflow/providers/amazon/aws/hooks/sagemaker_notebook.py: ## Review Comment: Refactored sagemaker_notebook.py into sagemaker.py. -- This is an automated

[GitHub] [airflow] erdos2n commented on pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n commented on PR #33408: URL: https://github.com/apache/airflow/pull/33408#issuecomment-1678909837 @eladkal , I added test cases for the `role_arn`, can you let me know if the tests were added correctly? Thank you -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] arjunanan6 commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
arjunanan6 commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678965803 Unfortunately, I do not know how to reproduce it yet. I've only seen this problem happen since the change to job_heartbeat_sec yesterday - but that was applied to all

[GitHub] [airflow] vincbeck merged pull request #33407: Fix outdated test name and description in BatchSensor

2023-08-15 Thread via GitHub
vincbeck merged PR #33407: URL: https://github.com/apache/airflow/pull/33407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] kaxil merged pull request #33296: add concrete examples for accessing context variables from TaskFlow tasks

2023-08-15 Thread via GitHub
kaxil merged PR #33296: URL: https://github.com/apache/airflow/pull/33296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] vincbeck commented on a diff in pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33408: URL: https://github.com/apache/airflow/pull/33408#discussion_r1294716332 ## airflow/providers/amazon/aws/operators/glue.py: ## @@ -53,7 +53,8 @@ class GlueJobOperator(BaseOperator): :param num_of_dpus: Number of AWS Glue DPUs to

[GitHub] [airflow] ephraimbuddy opened a new pull request, #33411: Fix getting correct commit from multiple referenced PR

2023-08-15 Thread via GitHub
ephraimbuddy opened a new pull request, #33411: URL: https://github.com/apache/airflow/pull/33411 When a PR is referenced by other PRs, our dev tool for getting the correct commit lists the latest commit when looking for the commit sha but we should get the oldest. If you run:

[GitHub] [airflow] fgalind1 commented on pull request #31066: Support naming customization on helm chart resources

2023-08-15 Thread via GitHub
fgalind1 commented on PR #31066: URL: https://github.com/apache/airflow/pull/31066#issuecomment-1679194150 does it looks good @jedcunningham /@potiuk?  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] ephraimbuddy opened a new pull request, #33412: Remove uploading to test pypi for releases

2023-08-15 Thread via GitHub
ephraimbuddy opened a new pull request, #33412: URL: https://github.com/apache/airflow/pull/33412 Test pypi doesn't work for org and our uploads are failing each time. Time to remove it! -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] ahidalgob opened a new pull request, #33413: Add read only validation to read only fields

2023-08-15 Thread via GitHub
ahidalgob opened a new pull request, #33413: URL: https://github.com/apache/airflow/pull/33413 Add read only validation to DagRunEditForm and TaskInstanceEditForm read only fields. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] tseruga opened a new issue, #33410: SlaMiss Records Never Created for Packaged DAGs

2023-08-15 Thread via GitHub
tseruga opened a new issue, #33410: URL: https://github.com/apache/airflow/issues/33410 ### Apache Airflow version 2.6.3 ### What happened First and foremost, I understand that the current posture is that the currently implemented SLA mechanism is buggy and is being

[GitHub] [airflow] o-nikolas closed pull request #33397: Remove `supports_ad_hoc_ti_run` from executor API

2023-08-15 Thread via GitHub
o-nikolas closed pull request #33397: Remove `supports_ad_hoc_ti_run` from executor API URL: https://github.com/apache/airflow/pull/33397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] alexbegg commented on pull request #33359: Add init_command parameter to MySqlHook

2023-08-15 Thread via GitHub
alexbegg commented on PR #33359: URL: https://github.com/apache/airflow/pull/33359#issuecomment-1679248385 @potiuk this is now ready for review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] ellisms commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
ellisms commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294857841 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1523,3 +1523,259 @@ def execute(self, context: Context) -> str: arn = ans["ExperimentArn"]

[GitHub] [airflow] ellisms commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
ellisms commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294874707 ## scripts/ci/docker-compose/empty/git_version: ## Review Comment: This should be gone now. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] insomnes opened a new issue, #33415: `set_state` error on task state not shared with DagRunState

2023-08-15 Thread via GitHub
insomnes opened a new issue, #33415: URL: https://github.com/apache/airflow/issues/33415 ### Apache Airflow version 2.6.3 ### What happened Got error while using `set_state()` to set task state to `TaskInstanceState.RESTARTING`

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1294751595 ## airflow/auth/managers/base_auth_manager.py: ## @@ -54,6 +58,67 @@ def get_user_id(self) -> str: def is_logged_in(self) -> bool: """Return whether

[GitHub] [airflow] erdos2n commented on a diff in pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n commented on code in PR #33408: URL: https://github.com/apache/airflow/pull/33408#discussion_r1294751897 ## airflow/providers/amazon/aws/hooks/glue.py: ## @@ -44,7 +44,8 @@ class GlueJobHook(AwsBaseHook): :param retry_limit: Maximum number of times to retry this

[GitHub] [airflow] erdos2n commented on a diff in pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n commented on code in PR #33408: URL: https://github.com/apache/airflow/pull/33408#discussion_r1294756676 ## airflow/providers/amazon/aws/hooks/glue.py: ## @@ -114,12 +119,14 @@ def create_glue_job_config(self) -> dict: "ScriptLocation":

[GitHub] [airflow] erdos2n commented on a diff in pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n commented on code in PR #33408: URL: https://github.com/apache/airflow/pull/33408#discussion_r1294781445 ## tests/providers/amazon/aws/hooks/test_glue.py: ## @@ -90,6 +90,56 @@ class JobNotFoundException(Exception): assert result is False

[GitHub] [airflow] erdos2n commented on a diff in pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n commented on code in PR #33408: URL: https://github.com/apache/airflow/pull/33408#discussion_r1294791349 ## tests/providers/amazon/aws/hooks/test_glue.py: ## @@ -66,6 +66,22 @@ def test_get_iam_execution_role(self, role_path): assert "Arn" in iam_role["Role"]

[GitHub] [airflow] o-nikolas commented on a diff in pull request #33397: Remove `supports_ad_hoc_ti_run` from executor API

2023-08-15 Thread via GitHub
o-nikolas commented on code in PR #33397: URL: https://github.com/apache/airflow/pull/33397#discussion_r1294821164 ## airflow/executors/base_executor.py: ## @@ -104,7 +104,6 @@ class BaseExecutor(LoggingMixin): :param parallelism: how many jobs should run at one time. Set

[GitHub] [airflow] ahidalgob commented on pull request #33413: Add read only validation to read only fields

2023-08-15 Thread via GitHub
ahidalgob commented on PR #33413: URL: https://github.com/apache/airflow/pull/33413#issuecomment-1679296127 It would be nice to have a base airflow form Class so we could modify `Form.populate_obj` there and not for every form that has a `ReadOnly` field, but I'm not sure the change it's

[airflow] branch main updated (a1576d39d2 -> 5b104a9020)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from a1576d39d2 Fix outdated test name and description in BatchSensor (#33407) add 5b104a9020 Fix getting

[GitHub] [airflow] ephraimbuddy merged pull request #33411: Fix getting correct commit from multiple referenced PR

2023-08-15 Thread via GitHub
ephraimbuddy merged PR #33411: URL: https://github.com/apache/airflow/pull/33411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] amoghrajesh commented on pull request #33379: Add `active_deadline_seconds` parameter to `KubernetesPodOperator`

2023-08-15 Thread via GitHub
amoghrajesh commented on PR #33379: URL: https://github.com/apache/airflow/pull/33379#issuecomment-1679270039 @hussein-awala seems like the commit didn't do the trick here. I pushed a new fix now, might just do the trick now -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] aamster commented on issue #33014: Clearing task from List Task Instance page in UI does not also clear downstream tasks?

2023-08-15 Thread via GitHub
aamster commented on issue #33014: URL: https://github.com/apache/airflow/issues/33014#issuecomment-1679297020 I would consider this a bug, not a new feature. I should never be uncertain what airflow is going to do. Here the obvious thing I would expect is that downstream tasks would be

[GitHub] [airflow] insomnes closed issue #33415: `set_state` error on task state not shared with DagRunState

2023-08-15 Thread via GitHub
insomnes closed issue #33415: `set_state` error on task state not shared with DagRunState URL: https://github.com/apache/airflow/issues/33415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] Lee-W opened a new pull request, #33403: Respect soft fail for core async sensors

2023-08-15 Thread via GitHub
Lee-W opened a new pull request, #33403: URL: https://github.com/apache/airflow/pull/33403 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] sonmezonur commented on a diff in pull request #33174: Add `pod_template_dict` field to `KubernetesPodOperator`

2023-08-15 Thread via GitHub
sonmezonur commented on code in PR #33174: URL: https://github.com/apache/airflow/pull/33174#discussion_r1294357934 ## tests/providers/cncf/kubernetes/operators/test_pod.py: ## @@ -920,6 +921,64 @@ def test_pod_template_file_kwargs_override(self, randomize_name,

[GitHub] [airflow] uranusjr opened a new pull request, #33404: Allow timetable to slightly miss catchup cutoff

2023-08-15 Thread via GitHub
uranusjr opened a new pull request, #33404: URL: https://github.com/apache/airflow/pull/33404 Previously, with catchup=False, CronTriggerTimetable would aggressively cut off a run if the scheduler doesn't ask to schedule the next run immediately. This causes DAGs to seemingly mysteriously

[GitHub] [airflow] Lee-W opened a new pull request, #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
Lee-W opened a new pull request, #33405: URL: https://github.com/apache/airflow/pull/33405 This is the following PR to https://github.com/apache/airflow/pull/33196 and intends to solve the issue that the soft_fail argument might not work on all sensors. This pull request addresses

[GitHub] [airflow] mis98zb commented on pull request #32876: enable db access for dag run listeners

2023-08-15 Thread via GitHub
mis98zb commented on PR #32876: URL: https://github.com/apache/airflow/pull/32876#issuecomment-1678742961 Thank you for the answer! But seems this does not work for me. The `@provide_session` will call submit() when the function exit, and it's inside the `@provide_session` of

[GitHub] [airflow] erdos2n opened a new pull request, #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
erdos2n opened a new pull request, #33408: URL: https://github.com/apache/airflow/pull/33408 GlueOperator currently requires `iam_role_name` and `endpoint_url` to fetch the ARN from AWS. This ARN eventually gets passed into the config for Glue Job. --- **^ Add

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294514377 ## airflow/providers/databricks/operators/databricks.py: ## @@ -24,6 +24,8 @@ from logging import Logger from typing import TYPE_CHECKING, Any, Sequence

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294513909 ## airflow/providers/databricks/operators/databricks.py: ## @@ -24,6 +24,8 @@ from logging import Logger from typing import TYPE_CHECKING, Any, Sequence

[GitHub] [airflow] dstandish commented on issue #21867: Retry for TaskGroup

2023-08-15 Thread via GitHub
dstandish commented on issue #21867: URL: https://github.com/apache/airflow/issues/21867#issuecomment-1678858953 Warning... this is a long one... just tried to catch up on the current state and share some thoughts and some reflections based on learnings in AIP-52 > We need also

[GitHub] [airflow] joshowen commented on pull request #33242: Index optimized fast path to avoid slower groupby queryplan

2023-08-15 Thread via GitHub
joshowen commented on PR #33242: URL: https://github.com/apache/airflow/pull/33242#issuecomment-1678691128 > I think we need a test case to cover the one DAG case. The one dag case is pretty well covered in the integration tests. I previously had a bug in this code and used those to

[GitHub] [airflow] dstandish commented on issue #21867: Retry for TaskGroup

2023-08-15 Thread via GitHub
dstandish commented on issue #21867: URL: https://github.com/apache/airflow/issues/21867#issuecomment-1678789067 Yeah it will be nice to have "living doc" of sorts, a draft AIP, where the proposal can be clearly outlined, edge cases and caveats noted, and we can refine and iterate. Will

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294515977 ## airflow/providers/databricks/operators/databricks.py: ## @@ -160,6 +162,142 @@ def get_link( return XCom.get_value(key=XCOM_RUN_PAGE_URL_KEY,

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294517299 ## airflow/providers/databricks/hooks/databricks.py: ## @@ -135,9 +137,27 @@ def __init__( ) -> None: super().__init__(databricks_conn_id,

[GitHub] [airflow] alexbegg commented on pull request #33359: Allow init_command in MySQL provider

2023-08-15 Thread via GitHub
alexbegg commented on PR #33359: URL: https://github.com/apache/airflow/pull/33359#issuecomment-1678600424 Moving to a draft until I rebase the branch and make it ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[airflow] branch main updated: openlineage: do not try to redact Proxy objects from deprecated config (#33393)

2023-08-15 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8e738cd0ad openlineage: do not try to redact

[GitHub] [airflow] mobuchowski merged pull request #33393: openlineage: do not try to redact Proxy objects from deprecated config

2023-08-15 Thread via GitHub
mobuchowski merged PR #33393: URL: https://github.com/apache/airflow/pull/33393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #33406: fix(TimeSensorAsync): use DAG timezone

2023-08-15 Thread via GitHub
boring-cyborg[bot] commented on PR #33406: URL: https://github.com/apache/airflow/pull/33406#issuecomment-1678741630 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] SapphicCode opened a new pull request, #33406: fix(TimeSensorAsync): use DAG timezone

2023-08-15 Thread via GitHub
SapphicCode opened a new pull request, #33406: URL: https://github.com/apache/airflow/pull/33406 Fixes: #33256 TimeSensorAsync used to convert naive timezones to UTC regardless of DAG timezone. This fixes that. Please let me know if this is a significant change and I'll write

[GitHub] [airflow] stikkireddy commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-08-15 Thread via GitHub
stikkireddy commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1294519821 ## airflow/providers/databricks/operators/databricks.py: ## @@ -160,6 +162,142 @@ def get_link( return XCom.get_value(key=XCOM_RUN_PAGE_URL_KEY,

[GitHub] [airflow] vincbeck commented on pull request #33408: [WIP] Issue ##33248 - GlueOperator IAM ARN

2023-08-15 Thread via GitHub
vincbeck commented on PR #33408: URL: https://github.com/apache/airflow/pull/33408#issuecomment-1679429681 You have some static checks failure, please [run them](https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst) to fix these failures. -- This is an automated message

[GitHub] [airflow] vincbeck commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294992827 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1523,3 +1524,246 @@ def execute(self, context: Context) -> str: arn = ans["ExperimentArn"]

[GitHub] [airflow] vincbeck commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294992979 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1523,3 +1524,246 @@ def execute(self, context: Context) -> str: arn = ans["ExperimentArn"]

[GitHub] [airflow] shahar1 commented on pull request #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
shahar1 commented on PR #33419: URL: https://github.com/apache/airflow/pull/33419#issuecomment-1679545805 It might be worth adding documentation of where/how it's being used to avoid (non-)accidental deletion for the next time :) -- This is an automated message from the Apache Git

[GitHub] [airflow] Dan-Git-Rep opened a new pull request, #33420: Update gcp.rst

2023-08-15 Thread via GitHub
Dan-Git-Rep opened a new pull request, #33420: URL: https://github.com/apache/airflow/pull/33420 three options to connect to GCP not the initial specified two --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] bbovenzi opened a new pull request, #33421: Improve graph nesting logic

2023-08-15 Thread via GitHub
bbovenzi opened a new pull request, #33421: URL: https://github.com/apache/airflow/pull/33421 Fix how we handle nested groups to make sure edge labels inside of groups are correctly positions and that nested nodes with the same label don't mess with rendering. | Before |

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115248 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295081179 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -81,3 +148,35 @@ def get_url_user_profile(self) -> str | None: if not

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115914 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from

[GitHub] [airflow] hussein-awala commented on a diff in pull request #33406: fix(TimeSensorAsync): use DAG timezone

2023-08-15 Thread via GitHub
hussein-awala commented on code in PR #33406: URL: https://github.com/apache/airflow/pull/33406#discussion_r1295148713 ## airflow/sensors/time_sensor.py: ## @@ -64,7 +64,7 @@ def __init__(self, *, target_time, **kwargs): self.target_time = target_time

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1294951943 ## airflow/auth/managers/base_auth_manager.py: ## @@ -54,6 +58,67 @@ def get_user_id(self) -> str: def is_logged_in(self) -> bool: """Return whether

[GitHub] [airflow] alexbegg opened a new pull request, #33416: Fixing status checks for FabAirflowSecurityManagerOverride

2023-08-15 Thread via GitHub
alexbegg opened a new pull request, #33416: URL: https://github.com/apache/airflow/pull/33416 This was filling pydocstyle rule D205 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] vincbeck commented on a diff in pull request #33219: Added Amazon SageMaker Notebook hook and operators

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33219: URL: https://github.com/apache/airflow/pull/33219#discussion_r1294989565 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1523,3 +1524,246 @@ def execute(self, context: Context) -> str: arn = ans["ExperimentArn"]

[GitHub] [airflow] jens-scheffler-bosch opened a new pull request, #33417: Add typing to concrete taskflow examples

2023-08-15 Thread via GitHub
jens-scheffler-bosch opened a new pull request, #33417: URL: https://github.com/apache/airflow/pull/33417 I just saw the PR #33296 in Slack and really liked it - I was struggeling to find concrete examples myself multiple times. Thanks to @RNHTTR for making this! When taking a look

[GitHub] [airflow] alexbegg commented on pull request #33416: Fixing status checks for FabAirflowSecurityManagerOverride

2023-08-15 Thread via GitHub
alexbegg commented on PR #33416: URL: https://github.com/apache/airflow/pull/33416#issuecomment-1679426859 @ferruzzi it looks like the PR for these files being added passed the status check before your PR changing adding rule D205, but then got merged after. This hopefully should fix

[airflow] branch main updated (5b104a9020 -> 3766ab07d7)

2023-08-15 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 5b104a9020 Fix getting correct commit from multiple referenced PR (#33411) add 3766ab07d7 More complete fix for

[GitHub] [airflow] ferruzzi commented on pull request #33416: Fixing static checks for FabAirflowSecurityManagerOverride

2023-08-15 Thread via GitHub
ferruzzi commented on PR #33416: URL: https://github.com/apache/airflow/pull/33416#issuecomment-1679501589 LGTM, I'll approve and merge it as soon as the CI goes green. Thanks for that. I made sure everything that was already committed passed, but there are bound to be a couple that

[GitHub] [airflow] potiuk opened a new pull request, #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
potiuk opened a new pull request, #33419: URL: https://github.com/apache/airflow/pull/33419 This reverts commit a29887451fa4807e56fb60e9ef72af4bcdf4924f. --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] potiuk merged pull request #33418: More complete fix for dev release scripts to filter commit for PR

2023-08-15 Thread via GitHub
potiuk merged PR #33418: URL: https://github.com/apache/airflow/pull/33418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
boring-cyborg[bot] commented on PR #33420: URL: https://github.com/apache/airflow/pull/33420#issuecomment-1679559348 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[airflow] branch main updated (3766ab07d7 -> cc360b73c9)

2023-08-15 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 3766ab07d7 More complete fix for dev release scripts to filter commit for PR (#33418) add cc360b73c9 Fixing

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115914 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from

[GitHub] [airflow] potiuk commented on a diff in pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33355: URL: https://github.com/apache/airflow/pull/33355#discussion_r1295118289 ## airflow/operators/python.py: ## @@ -606,7 +613,39 @@ def _prepare_venv(self, venv_path: Path) -> None: index_urls=self.index_urls, ) +

[GitHub] [airflow] hussein-awala commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
hussein-awala commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1679598748 Actually we set the resource version to 0 on failure: ```python except Exception: self.log.exception("Unknown error in

[GitHub] [airflow] hussein-awala commented on a diff in pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
hussein-awala commented on code in PR #33420: URL: https://github.com/apache/airflow/pull/33420#discussion_r1295123482 ## docs/apache-airflow-providers-google/connections/gcp.rst: ## @@ -27,7 +27,7 @@ The Google Cloud connection type enables the Google Cloud Integrations.

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295119967 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[GitHub] [airflow] hussein-awala merged pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
hussein-awala merged PR #33420: URL: https://github.com/apache/airflow/pull/33420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

  1   2   >