[GitHub] [airflow] maxnathaniel commented on pull request #28953: Updated Telegram Provider to ensure compatbility with >=20.0.0

2023-01-19 Thread GitBox
maxnathaniel commented on PR #28953: URL: https://github.com/apache/airflow/pull/28953#issuecomment-1398028299 @Taragolis Sorry have been busy past couple of days. Do you think I should go ahead to replace `python-telegram-bot` with

[GitHub] [airflow] ecerulm commented on a diff in pull request #29016: Fix leak sensitive field via V1EnvVar on exception

2023-01-19 Thread GitBox
ecerulm commented on code in PR #29016: URL: https://github.com/apache/airflow/pull/29016#discussion_r1082180416 ## airflow/utils/log/secrets_masker.py: ## @@ -200,10 +222,18 @@ def _redact(self, item: Redactable, name: str | None, depth: int) -> Redacted: if name

[GitHub] [airflow] phanikumv commented on a diff in pull request #29014: Add deferrable mode to `DbtCloudRunJobOperator`

2023-01-19 Thread GitBox
phanikumv commented on code in PR #29014: URL: https://github.com/apache/airflow/pull/29014#discussion_r1082167660 ## docs/apache-airflow-providers-dbt-cloud/operators.rst: ## @@ -35,10 +35,10 @@ Trigger a dbt Cloud Job Use the

[GitHub] [airflow] potiuk commented on issue #29031: Scheduler pod restarts

2023-01-19 Thread GitBox
potiuk commented on issue #29031: URL: https://github.com/apache/airflow/issues/29031#issuecomment-1397996468 Hard to say. Maybe you have REALLY SLOW network or filesystem mounted in your system? Or maybe your docker engine has limited memory (as it happens on MacOS where we recommend to

[GitHub] [airflow] Subhashini2610 commented on issue #29031: Scheduler pod restarts

2023-01-19 Thread GitBox
Subhashini2610 commented on issue #29031: URL: https://github.com/apache/airflow/issues/29031#issuecomment-1397974322 @hussein-awala Thanks for the insight. I have only one DAG (hello_world.py) synced, which is also not scheduled for periodic runs. Also, I have checked the pod metrics. The

[GitHub] [airflow] tirkarthi commented on pull request #28256: Include full path to Python files under zip path while clearing import errors.

2023-01-19 Thread GitBox
tirkarthi commented on PR #28256: URL: https://github.com/apache/airflow/pull/28256#issuecomment-1397945755 @potiuk @uranusjr Sorry, I am little confused. Currently `ImportError` model has filename column that stores full path to Python along with zipfile like

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-01-19 Thread GitBox
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1082091328 ## airflow/cli/cli_parser.py: ## @@ -17,2165 +17,35 @@ # specific language governing permissions and limitations # under the License. """Command-line

[GitHub] [airflow] schwartzpub commented on issue #29037: Cron schedule and Time Zones leads to incorrect intervals

2023-01-19 Thread GitBox
schwartzpub commented on issue #29037: URL: https://github.com/apache/airflow/issues/29037#issuecomment-1397824977 Last run/next run aside -- what other details are needed that aren't provided above? For reference, the documentation for 2.5 is where the recommendation for

[GitHub] [airflow] potiuk commented on pull request #28693: AIP-44 Migrate DagModel.get_paused_dag_ids to Internal API

2023-01-19 Thread GitBox
potiuk commented on PR #28693: URL: https://github.com/apache/airflow/pull/28693#issuecomment-1397793451 > Is there any action item/comment here I should act on or we can forward with this PR? Rebasing/conflict resolving after we merged #28841 -- This is an automated message from

[GitHub] [airflow] potiuk commented on a diff in pull request #28618: Add deferrable mode to DataprocInstantiateWorkflowTemplateOperator

2023-01-19 Thread GitBox
potiuk commented on code in PR #28618: URL: https://github.com/apache/airflow/pull/28618#discussion_r1082013189 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -84,3 +84,73 @@ async def run(self): raise AirflowException(f"Dataproc job

[GitHub] [airflow] potiuk commented on pull request #28956: Move project and license docs down to start with developer-focused docs

2023-01-19 Thread GitBox
potiuk commented on PR #28956: URL: https://github.com/apache/airflow/pull/28956#issuecomment-1397788465 Oh Absolutely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[airflow] branch main updated (e7b21ed23c -> d03b9a7691)

2023-01-19 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e7b21ed23c nice table at the end of system test execs (#29051) add d03b9a7691 Move project and license docs down

[GitHub] [airflow] potiuk merged pull request #28956: Move project and license docs down to start with developer-focused docs

2023-01-19 Thread GitBox
potiuk merged PR #28956: URL: https://github.com/apache/airflow/pull/28956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] hussein-awala commented on issue #29031: Scheduler pod restarts

2023-01-19 Thread GitBox
hussein-awala commented on issue #29031: URL: https://github.com/apache/airflow/issues/29031#issuecomment-1397785578 Since the liveness probe request return `503`, you scheduler could be overloaded, and there are several different causes for this problem: - The request/limit resources

[GitHub] [airflow] potiuk commented on issue #19781: A way to test email configuration

2023-01-19 Thread GitBox
potiuk commented on issue #19781: URL: https://github.com/apache/airflow/issues/19781#issuecomment-1397782458 > btw should i create a separate ticket on that or will just a PR suffice No need for issues in Airflow. PR is enough in vast majority of canses. -- This is an automated

[airflow] branch main updated (17e8bb7f9e -> e7b21ed23c)

2023-01-19 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 17e8bb7f9e Update provide_bucket_name() decorator to handle new conn_type (#28706) add e7b21ed23c nice table at

[GitHub] [airflow] potiuk merged pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
potiuk merged PR #29051: URL: https://github.com/apache/airflow/pull/29051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] o-nikolas opened a new pull request, #29055: Onikolas/aip 51/executor cli vending

2023-01-19 Thread GitBox
o-nikolas opened a new pull request, #29055: URL: https://github.com/apache/airflow/pull/29055 This PR addresses executor coupling **5a** and **5b** in Airflow CLI. See [this issue](https://github.com/apache/airflow/issues/27932) (#27932) for more context. ## Please read before

[GitHub] [airflow] potiuk commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
potiuk commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397779289 > > (and we should wait for it). > > You mean we should not merge this until AIP-52 gives us a way to mark teardown tasks ? Nope. Wait with separating the tables. I think

[GitHub] [airflow] potiuk commented on issue #28919: Airflow API kerberos authentication error

2023-01-19 Thread GitBox
potiuk commented on issue #28919: URL: https://github.com/apache/airflow/issues/28919#issuecomment-139007 How do you expect your kerberos user to map to Airflow @BMFH? I believe indeed the current way kerberos user is retrieved is wrong. From what I see. the user that is set by

[GitHub] [airflow] potiuk opened a new pull request, #29054: [Experimental] Treat kerberos identity as email in authentication

2023-01-19 Thread GitBox
potiuk opened a new pull request, #29054: URL: https://github.com/apache/airflow/pull/29054 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] vandonr-amz commented on pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-19 Thread GitBox
vandonr-amz commented on PR #28816: URL: https://github.com/apache/airflow/pull/28816#issuecomment-1397776181 @eladkal @uranusjr anything else you'd like changed with that PR ? Otherwise we could merge it  -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] vandonr-amz opened a new pull request, #29053: introduce base class for EKS sensors

2023-01-19 Thread GitBox
vandonr-amz opened a new pull request, #29053: URL: https://github.com/apache/airflow/pull/29053 most of the EKS sensors do the same thing, this can be grouped in one base class, a bit like what was done here

[GitHub] [airflow] hussein-awala commented on issue #29002: KubernetesPodOperator xcom push failure

2023-01-19 Thread GitBox
hussein-awala commented on issue #29002: URL: https://github.com/apache/airflow/issues/29002#issuecomment-1397773389 I have the same problem with some of my pod tasks, I just created this [PR](https://github.com/apache/airflow/pull/29052), can you check it please? -- This is an automated

[GitHub] [airflow] hussein-awala opened a new pull request, #29052: Fix KubernetesPodOperator xcom push when `get_logs=False`

2023-01-19 Thread GitBox
hussein-awala opened a new pull request, #29052: URL: https://github.com/apache/airflow/pull/29052 closes: #29002 --- **^ Add meaningful description above** When `get_logs=False`, we should wait the `base` container completion before reading the xcom file, to do

[GitHub] [airflow] o-nikolas commented on pull request #28934: Remove hard-coded executor-database coupling

2023-01-19 Thread GitBox
o-nikolas commented on PR #28934: URL: https://github.com/apache/airflow/pull/28934#issuecomment-1397769679 Hey @utkarsharma2! Have you had any time to revisit this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[airflow] branch main updated (1ab7ea81a1 -> 17e8bb7f9e)

2023-01-19 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1ab7ea81a1 uniformize getting hook through cached property in aws sensors (#29001) add 17e8bb7f9e Update

[GitHub] [airflow] o-nikolas merged pull request #28706: Update provide_bucket_name() decorator to handle new conn_type

2023-01-19 Thread GitBox
o-nikolas merged PR #28706: URL: https://github.com/apache/airflow/pull/28706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #27156: Add documentation for BigQuery transfer operators

2023-01-19 Thread GitBox
github-actions[bot] commented on PR #27156: URL: https://github.com/apache/airflow/pull/27156#issuecomment-1397768482 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] github-actions[bot] commented on pull request #28056: Move GCP Sheets system tests

2023-01-19 Thread GitBox
github-actions[bot] commented on PR #28056: URL: https://github.com/apache/airflow/pull/28056#issuecomment-1397768424 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] vandonr-amz commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
vandonr-amz commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397764394 > (and we should wait for it). You mean we should not merge this until AIP-52 gives us a way to mark teardown tasks ? -- This is an automated message from the Apache Git

[airflow] branch main updated: uniformize getting hook through cached property in aws sensors (#29001)

2023-01-19 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1ab7ea81a1 uniformize getting hook through

[GitHub] [airflow] o-nikolas merged pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-19 Thread GitBox
o-nikolas merged PR #29001: URL: https://github.com/apache/airflow/pull/29001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #28902: Chart: Update default git-sync version to 3.6.2

2023-01-19 Thread GitBox
potiuk commented on PR #28902: URL: https://github.com/apache/airflow/pull/28902#issuecomment-1397758005 @aleveille - still working on it ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk commented on issue #18010: Airflow scheduler with statsd enabled crashes when dag_id contains unexpected characters

2023-01-19 Thread GitBox
potiuk commented on issue #18010: URL: https://github.com/apache/airflow/issues/18010#issuecomment-1397756161 Should be part of discussion here: https://lists.apache.org/thread/96tco8dfs4mh12kqm1pwjhm5mqr54qbm . I think we should limit ID to ASCII and add extra description. -- This is

[GitHub] [airflow] potiuk commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
potiuk commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397754758 > Is the watcher supposed to be the "separator" ? Because in that case, disable_job_queue is part of the teardown and apparently executed before the watcher (I sort by end_date, maybe

[GitHub] [airflow] vandonr-amz commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
vandonr-amz commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397739480 > Can we maybe also split the watcher and what happens after to a separate table (indicating this is a teardown?). We have enough of a convention to know it I think and that would

[GitHub] [airflow] vchiapaikeo commented on a diff in pull request #28942: Fix @task.kubernetes to receive input and send output

2023-01-19 Thread GitBox
vchiapaikeo commented on code in PR #28942: URL: https://github.com/apache/airflow/pull/28942#discussion_r1081969377 ## airflow/providers/cncf/kubernetes/decorators/kubernetes.py: ## @@ -82,11 +83,42 @@ def _get_python_source(self): res = remove_task_decorator(res,

[GitHub] [airflow] potiuk commented on issue #28880: Can't configure Kubernetes and Celery workers in Helm Chart

2023-01-19 Thread GitBox
potiuk commented on issue #28880: URL: https://github.com/apache/airflow/issues/28880#issuecomment-1397737670 Why not - if someoene would like to pick it, i marked it as good first issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] Taragolis commented on pull request #29045: Move imports in AWS SqlToS3Operator transfer to callable function

2023-01-19 Thread GitBox
Taragolis commented on PR #29045: URL: https://github.com/apache/airflow/pull/29045#issuecomment-1397736154 @joarobles BTW, did you check is this changes reduce DAG Processing time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] potiuk commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
potiuk commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397736179 (BTW. Later when we will have explicit setup/teardown support in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks - we will also be able to

[GitHub] [airflow] potiuk commented on pull request #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
potiuk commented on PR #29051: URL: https://github.com/apache/airflow/pull/29051#issuecomment-1397735057 Can we maybe also split the watcher and what happens after to a separate table (indicating this is a teardown?) we have a enough of a convention to know it I think and that would be

[GitHub] [airflow] vandonr-amz opened a new pull request, #29051: Add a summary table at the end of system test execs

2023-01-19 Thread GitBox
vandonr-amz opened a new pull request, #29051: URL: https://github.com/apache/airflow/pull/29051 In my experience, it's currently hard to see exactly what error caused an example dag/system test to fail, because it doesn't stop right away, and teardown tasks can error as well, so it

[GitHub] [airflow] bugraoz93 commented on a diff in pull request #28795: Migrate Models Variable to Internal API

2023-01-19 Thread GitBox
bugraoz93 commented on code in PR #28795: URL: https://github.com/apache/airflow/pull/28795#discussion_r1081962687 ## airflow/models/variable.py: ## @@ -181,6 +183,7 @@ def set( @classmethod Review Comment: Definitely :) -- This is an automated message from the

[GitHub] [airflow] o-nikolas commented on pull request #28528: Fixes to how DebugExecutor handles sensors

2023-01-19 Thread GitBox
o-nikolas commented on PR #28528: URL: https://github.com/apache/airflow/pull/28528#issuecomment-1397726771 > > > Not sure - after all the discussions and other things happening what would be the best course of action here. > > > > > > I think the course of action is to wait for

[GitHub] [airflow] potiuk commented on pull request #28528: Fixes to how DebugExecutor handles sensors

2023-01-19 Thread GitBox
potiuk commented on PR #28528: URL: https://github.com/apache/airflow/pull/28528#issuecomment-1397725857 > > Not sure - after all the discussions and other things happening what would be the best course of action here. > > I think the course of action is to wait for an

[GitHub] [airflow] potiuk commented on a diff in pull request #28795: Migrate Models Variable to Internal API

2023-01-19 Thread GitBox
potiuk commented on code in PR #28795: URL: https://github.com/apache/airflow/pull/28795#discussion_r1081960057 ## airflow/models/variable.py: ## @@ -181,6 +183,7 @@ def set( @classmethod Review Comment: Then also `cls` has to be removed. -- This is an automated

[GitHub] [airflow] vandonr-amz commented on pull request #28528: Fixes to how DebugExecutor handles sensors

2023-01-19 Thread GitBox
vandonr-amz commented on PR #28528: URL: https://github.com/apache/airflow/pull/28528#issuecomment-1397724723 > Not sure - after all the discussions and other things happening what would be the best course of action here. I think the course of action is to wait for an

[GitHub] [airflow] potiuk commented on issue #28847: Graph UI: Add Filter Downstream & Filter DownStream & Upstream

2023-01-19 Thread GitBox
potiuk commented on issue #28847: URL: https://github.com/apache/airflow/issues/28847#issuecomment-1397724294 Feel free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] bugraoz93 commented on a diff in pull request #28795: Migrate Models Variable to Internal API

2023-01-19 Thread GitBox
bugraoz93 commented on code in PR #28795: URL: https://github.com/apache/airflow/pull/28795#discussion_r1081958464 ## airflow/models/variable.py: ## @@ -181,6 +183,7 @@ def set( @classmethod Review Comment: My bad. Thanks for pointing it out! It should be

[GitHub] [airflow] potiuk commented on pull request #28528: Fixes to how DebugExecutor handles sensors

2023-01-19 Thread GitBox
potiuk commented on PR #28528: URL: https://github.com/apache/airflow/pull/28528#issuecomment-1397721348 Not sure - after all the discussions and other things happening what would be the best course of action here. @o-nikolas - maybe you can make a call (especially in relation to AIP-51.

[GitHub] [airflow] potiuk commented on pull request #28822: Disable GitSync + Persistence combo in the Helm Chart

2023-01-19 Thread GitBox
potiuk commented on PR #28822: URL: https://github.com/apache/airflow/pull/28822#issuecomment-1397718391 Still thinking on it :). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk commented on issue #17977: DagRun marked as success without executing task

2023-01-19 Thread GitBox
potiuk commented on issue #17977: URL: https://github.com/apache/airflow/issues/17977#issuecomment-1397715678 > First, we understand the sorry reality of OSS development in which there's this huge number of entitled, mostly corporate users that demand updates, plans and SLA's from a

[GitHub] [airflow] yxiao1996 commented on issue #28847: Graph UI: Add Filter Downstream & Filter DownStream & Upstream

2023-01-19 Thread GitBox
yxiao1996 commented on issue #28847: URL: https://github.com/apache/airflow/issues/28847#issuecomment-1397714573 looks like an interesting idea, may I take a shot at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] notatallshaw-gts commented on issue #29037: Cron schedule and Time Zones leads to incorrect intervals

2023-01-19 Thread GitBox
notatallshaw-gts commented on issue #29037: URL: https://github.com/apache/airflow/issues/29037#issuecomment-1397703474 Be aware that the "Last Run" and "Next Run" are for Airflow's logical_date/execution_date which can be a little confusing to interpret at times:

[GitHub] [airflow] potiuk closed issue #28857: unsuspected behavior on execution_timeout for KubernetesPodOperator

2023-01-19 Thread GitBox
potiuk closed issue #28857: unsuspected behavior on execution_timeout for KubernetesPodOperator URL: https://github.com/apache/airflow/issues/28857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk commented on issue #28857: unsuspected behavior on execution_timeout for KubernetesPodOperator

2023-01-19 Thread GitBox
potiuk commented on issue #28857: URL: https://github.com/apache/airflow/issues/28857#issuecomment-1397687057 Can you please upgrede to latest version provieded by MWAA please? There were a complete rewrite of K8S interface since 2.2.2 becaus of similar problems you experienced. The only

[GitHub] [airflow] potiuk commented on a diff in pull request #28795: Migrate Models Variable to Internal API

2023-01-19 Thread GitBox
potiuk commented on code in PR #28795: URL: https://github.com/apache/airflow/pull/28795#discussion_r1081924148 ## airflow/models/variable.py: ## @@ -181,6 +183,7 @@ def set( @classmethod Review Comment: Why not @staticmethod here (As opposed to set above)? --

[GitHub] [airflow] potiuk commented on issue #22438: Migrate Amazon example DAGs to new design

2023-01-19 Thread GitBox
potiuk commented on issue #22438: URL: https://github.com/apache/airflow/issues/22438#issuecomment-1397677969 > If I'm not mistaken, that would require changes to the system test runner which I think currently triggers on the "example" prefix. This was a holdout to when they were still

[airflow] branch main updated: AIP-44 Initialize methods map for Internal API RPC endpoint in the method (#28841)

2023-01-19 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d0473a1357 AIP-44 Initialize methods map for

[GitHub] [airflow] potiuk merged pull request #28841: AIP-44 Initialize methods map for Internal API RPC endpoint in the method

2023-01-19 Thread GitBox
potiuk merged PR #28841: URL: https://github.com/apache/airflow/pull/28841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] sean-rose commented on issue #29049: Recursively cleared external task sensors using reschedule mode instantly time out if previous run is older than sensor timeout

2023-01-19 Thread GitBox
sean-rose commented on issue #29049: URL: https://github.com/apache/airflow/issues/29049#issuecomment-1397653484 > when recursively clearing tasks that won't work properly for tasks in downstream DAGs It's worth noting that's there's also a potentially worse outcome than sensors

[GitHub] [airflow] sean-rose opened a new issue, #29049: Recursively cleared external task sensors using reschedule mode instantly time out if previous run is older than sensor timeout

2023-01-19 Thread GitBox
sean-rose opened a new issue, #29049: URL: https://github.com/apache/airflow/issues/29049 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Using Airflow 2.3.3, when recursively clearing downstream tasks any cleared external

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29049: Recursively cleared external task sensors using reschedule mode instantly time out if previous run is older than sensor timeout

2023-01-19 Thread GitBox
boring-cyborg[bot] commented on issue #29049: URL: https://github.com/apache/airflow/issues/29049#issuecomment-1397649408 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] schwartzpub commented on issue #29037: Cron schedule and Time Zones leads to incorrect intervals

2023-01-19 Thread GitBox
schwartzpub commented on issue #29037: URL: https://github.com/apache/airflow/issues/29037#issuecomment-1397643641 Looking through the documentation I see that it is recommended to keep the airflow cfg at UTC. Having set this back to UTC and reimporting the DAG which has been made TZ

[GitHub] [airflow] Taragolis commented on pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-19 Thread GitBox
Taragolis commented on PR #29001: URL: https://github.com/apache/airflow/pull/29001#issuecomment-1397632969 > One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it. I think ideally we should pass every

[GitHub] [airflow] Taragolis commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-19 Thread GitBox
Taragolis commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1081871586 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] Taragolis commented on a diff in pull request #29038: Add HttpHookAsync for deferrable implementation

2023-01-19 Thread GitBox
Taragolis commented on code in PR #29038: URL: https://github.com/apache/airflow/pull/29038#discussion_r1081712597 ## airflow/providers/http/hooks/http.py: ## @@ -246,3 +253,142 @@ def test_connection(self): return True, "Connection successfully tested"

[GitHub] [airflow] josh-fell opened a new pull request, #29048: Allow downloading of dbt Cloud artifacts to non-existent paths

2023-01-19 Thread GitBox
josh-fell opened a new pull request, #29048: URL: https://github.com/apache/airflow/pull/29048 Closes: #27107 Currently the DbtCloudGetJobRunArtifactOperator does not support downloading dbt Cloud job run artifacts to non-existent local paths. This means users couldn't specify a

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081839251 ## airflow/providers/cncf/kubernetes/hooks/kubernetes.py: ## @@ -16,19 +16,28 @@ # under the License. from __future__ import annotations +import contextlib import

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081838952 ## airflow/providers/cncf/kubernetes/hooks/kubernetes.py: ## @@ -16,19 +16,28 @@ # under the License. from __future__ import annotations +import contextlib import

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081838371 ## airflow/providers/cncf/kubernetes/hooks/kubernetes.py: ## @@ -431,3 +446,150 @@ def _get_bool(val) -> bool | None: elif val.strip().lower() == "false":

[GitHub] [airflow] kaxil commented on pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on PR #29017: URL: https://github.com/apache/airflow/pull/29017#issuecomment-1397570021 @dstandish can you review this PR please since you last touched KPO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081836212 ## airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py: ## @@ -509,48 +535,154 @@ def execute(self, context: Context): if self.do_xcom_push:

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081832268 ## airflow/providers/cncf/kubernetes/hooks/kubernetes.py: ## @@ -431,3 +446,150 @@ def _get_bool(val) -> bool | None: elif val.strip().lower() == "false":

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081831491 ## airflow/providers/cncf/kubernetes/hooks/kubernetes.py: ## @@ -431,3 +446,150 @@ def _get_bool(val) -> bool | None: elif val.strip().lower() == "false":

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081827405 ## generated/provider_dependencies.json: ## @@ -209,8 +209,10 @@ "cncf.kubernetes": { "deps": [ "apache-airflow>=2.3.0", + "asgiref>=3.5.2",

[GitHub] [airflow] kaxil commented on a diff in pull request #29017: Add deferrable mode to KubernetesPodOperator

2023-01-19 Thread GitBox
kaxil commented on code in PR #29017: URL: https://github.com/apache/airflow/pull/29017#discussion_r1081824699 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -1,19 +1,19 @@ .. Licensed to the Apache Software Foundation (ASF) under one -or more

[GitHub] [airflow] kaxil commented on a diff in pull request #29014: Add deferrable mode to `DbtCloudRunJobOperator`

2023-01-19 Thread GitBox
kaxil commented on code in PR #29014: URL: https://github.com/apache/airflow/pull/29014#discussion_r1081817181 ## docs/apache-airflow-providers-dbt-cloud/operators.rst: ## @@ -35,10 +35,10 @@ Trigger a dbt Cloud Job Use the

[GitHub] [airflow] sunank200 opened a new pull request, #29047: Add Livy Operator with deferrable mode

2023-01-19 Thread GitBox
sunank200 opened a new pull request, #29047: URL: https://github.com/apache/airflow/pull/29047 This PR donates the following `LivyOperator` with a deferrable mode that wraps the Apache Livy batch REST API, allowing submission of a Spark application to the underlying cluster

[GitHub] [airflow] rkarish commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
rkarish commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1397517257 Sounds good, I will investigate tonight when I have time and reach out in Slack if I need any assistance. Thanks Jarek. -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] inytar commented on issue #28973: Dynamic Task Mapping skips tasks before upstream has started

2023-01-19 Thread GitBox
inytar commented on issue #28973: URL: https://github.com/apache/airflow/issues/28973#issuecomment-1397505498 > @inytar , it works for me in the current main. No skipping. What are your other settings like? I've seen the issue on different settings/environments: - Production

[GitHub] [airflow] potiuk commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
potiuk commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-139741 But if you are completely lost after rebase and checking other PRs and cannot figure how your change caused it, you can always ask for help. Sometimes there are weird, indirect chain

[GitHub] [airflow] potiuk commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
potiuk commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1397498135 > Thanks Jarek, and please let me know if there is anything else that I can do for this, like additional documentation or example code. Also, per the failing static checks and tests, is

[GitHub] [airflow] potiuk commented on issue #29031: Scheduler pod restarts

2023-01-19 Thread GitBox
potiuk commented on issue #29031: URL: https://github.com/apache/airflow/issues/29031#issuecomment-1397495433 > Same problem, worker pod restarting without reason, airflow 2.2.5 chart version 1.6.0 How do you know it's the same problem @cguermi ? Even the original report does not

[GitHub] [airflow] potiuk commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
potiuk commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1397486576 Yeah. I like to unify the tests and make our tests `pytest-only`. Great that we can disable the built-in plugin to verify it (and to prevent it from accidentally happen in the future).

[GitHub] [airflow] potiuk commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
potiuk commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1397487058 (@uranusjr ?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk commented on pull request #27829: Improving the release process

2023-01-19 Thread GitBox
potiuk commented on PR #27829: URL: https://github.com/apache/airflow/pull/27829#issuecomment-1397483090 I think you might wan to re-install breeze (via the composite action we use elsewhere) just before "Fix ownership" job. This should fix the last problem @ephraimbuddy -- This is an

[GitHub] [airflow] maggesssss commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-19 Thread GitBox
magges commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1081696287 ## airflow/providers/amazon/aws/example_dags/example_s3_to_sql.py: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review

[GitHub] [airflow] maggesssss commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-19 Thread GitBox
magges commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1081694789 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] potiuk commented on pull request #28459: Migrate remaining Always & CLI tests to `pytest`

2023-01-19 Thread GitBox
potiuk commented on PR #28459: URL: https://github.com/apache/airflow/pull/28459#issuecomment-1397453882 LGTM. @dimberman -> do you still have doubts? Or should we merge this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] vandonr-amz commented on pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-19 Thread GitBox
vandonr-amz commented on PR #29001: URL: https://github.com/apache/airflow/pull/29001#issuecomment-1397443105 > One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it. yes I'd rather keep this PR well

[GitHub] [airflow] RachitSharma2001 commented on pull request #28706: Update provide_bucket_name() decorator to handle new conn_type

2023-01-19 Thread GitBox
RachitSharma2001 commented on PR #28706: URL: https://github.com/apache/airflow/pull/28706#issuecomment-1397437544 Sounds good! I have committed those suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] rkarish commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
rkarish commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1397437288 Thanks Jarek, and please let me know if there is anything else that I can do for this, like additional documentation or example code. Also, per the failing static checks and tests, is

[GitHub] [airflow] potiuk commented on issue #27285: Using .output on non-templated fields

2023-01-19 Thread GitBox
potiuk commented on issue #27285: URL: https://github.com/apache/airflow/issues/27285#issuecomment-1397432952 > Hey Jarek, I've opened a PR for this. I took a slightly different approach than a DAG level setting. I managed the functionality at the Operator level, but this can still be

[GitHub] [airflow] potiuk commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
potiuk commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1397432109 (And tests and static checks need to be fixed BTW). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
potiuk commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1397431380 cc: @uranusjr - any particular concerns for that one (especially for mapped operators ?). I think we might need some specific exclusions ? -- This is an automated message from the

[GitHub] [airflow] potiuk commented on pull request #28999: Prepare ad hoc provider release for Docker and Cassandra

2023-01-19 Thread GitBox
potiuk commented on PR #28999: URL: https://github.com/apache/airflow/pull/28999#issuecomment-1397428045 Note. I do not think we **need** to release Cassandra (but if you want - I am ok with it too). The fact that eventlet team quickly fixed the problem, made the problem far less. of a

[GitHub] [airflow] kolfild26 commented on a diff in pull request #28998: write to DB a user who triggered a dag via API endpoint request

2023-01-19 Thread GitBox
kolfild26 commented on code in PR #28998: URL: https://github.com/apache/airflow/pull/28998#discussion_r1081657326 ## airflow/api_connexion/endpoints/dag_run_endpoint.py: ## @@ -280,6 +285,12 @@ def get_dag_runs_batch(*, session: Session = NEW_SESSION) -> APIResponse: ],

[GitHub] [airflow] kolfild26 commented on a diff in pull request #28998: write to DB a user who triggered a dag via API endpoint request

2023-01-19 Thread GitBox
kolfild26 commented on code in PR #28998: URL: https://github.com/apache/airflow/pull/28998#discussion_r1081624869 ## airflow/api_connexion/endpoints/dag_run_endpoint.py: ## @@ -319,6 +330,19 @@ def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:

  1   2   >