[GitHub] [airflow] rafalh commented on a diff in pull request #28970: Fix support for macros with dots in DataProcJobBuilder

2023-01-19 Thread GitBox
rafalh commented on code in PR #28970: URL: https://github.com/apache/airflow/pull/28970#discussion_r1081250709 ## tests/providers/google/cloud/hooks/test_dataproc.py: ## @@ -1006,17 +1006,9 @@ def test_set_python_main(self): self.builder.set_python_main(main)

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #28998: write to DB a user who triggered a dag via API endpoint request

2023-01-19 Thread GitBox
ephraimbuddy commented on code in PR #28998: URL: https://github.com/apache/airflow/pull/28998#discussion_r1081247196 ## airflow/api_connexion/endpoints/dag_run_endpoint.py: ## @@ -319,6 +330,19 @@ def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:

[GitHub] [airflow] sunank200 opened a new pull request, #29038: Add HttpHookAsync for deferrable implementation

2023-01-19 Thread GitBox
sunank200 opened a new pull request, #29038: URL: https://github.com/apache/airflow/pull/29038 This PR donates the following `HttpHookAsync` that interacts with HTTP servers using Python Async. This was developed in [astronomer-providers](https://github.com/astronomer/astronomer-provid

[GitHub] [airflow] ephraimbuddy commented on issue #28973: Dynamic Task Mapping skips tasks before upstream has started

2023-01-19 Thread GitBox
ephraimbuddy commented on issue #28973: URL: https://github.com/apache/airflow/issues/28973#issuecomment-1396958328 @inytar , it works for me in the current main. No skipping. What are your other settings like? @bbovenzi , it doesn't come up on the graph view in the current main

[GitHub] [airflow] Taragolis commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
Taragolis commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1396941580 @uranusjr In additional we could add `-p no:nose` in pytest.ini adoptions, which disable autouse `setup()` and `teardown()` methods. WDYT? -- This is an automated message fr

[GitHub] [airflow] schwartzpub opened a new issue, #29037: Cron schedule and Time Zones leads to incorrect intervals

2023-01-19 Thread GitBox
schwartzpub opened a new issue, #29037: URL: https://github.com/apache/airflow/issues/29037 ### Apache Airflow version 2.5.0 ### What happened When using cron syntax for DAG schedule, the scheduler is not running DAGs at the correct time for my timezone. For instance, a

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29037: Cron schedule and Time Zones leads to incorrect intervals

2023-01-19 Thread GitBox
boring-cyborg[bot] commented on issue #29037: URL: https://github.com/apache/airflow/issues/29037#issuecomment-1396932423 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] Taragolis commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
Taragolis commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1396928177 Yep, check in general this actually work (except teardown fixture) ```python import pytest class TestNoseSetupTearDown: def setup(self): self.

[GitHub] [airflow] Taragolis commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
Taragolis commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1396909639 Well even if this work then at least this feature undocumented in `pytest`, only this methods [documented](https://docs.pytest.org/en/latest/how-to/xunit_setup.html) -- This is an

[GitHub] [airflow] Taragolis commented on issue #29036: Top level code imports in AWS transfer

2023-01-19 Thread GitBox
Taragolis commented on issue #29036: URL: https://github.com/apache/airflow/issues/29036#issuecomment-1396898823 Fill free to make a PR. I would also recommend check all other providers is it load or not `pandas` or `numpy` in the Top Level and also fix other cases -- This is an autom

[GitHub] [airflow] Taragolis commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-19 Thread GitBox
Taragolis commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396890991 Seems like now as much stable as match possible. I add different combination for create separate process which spawn LocalTaskJob: * fork * fork daemonized * spawn * spa

[GitHub] [airflow] uranusjr commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
uranusjr commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1396879866 These should continue to work after they remove Nose compatibility so I think this PR is unnecessary. -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [airflow] Taragolis commented on pull request #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
Taragolis commented on PR #29035: URL: https://github.com/apache/airflow/pull/29035#issuecomment-1396860105 And after found about deprecation functionality in [pytest's changelog](https://docs.pytest.org/en/latest/changelog.html) I finally realise why test which use setup/teardown methods w

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29036: Top level code imports in AWS transfer

2023-01-19 Thread GitBox
boring-cyborg[bot] commented on issue #29036: URL: https://github.com/apache/airflow/issues/29036#issuecomment-1396853878 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] joarobles opened a new issue, #29036: Top level code imports in AWS transfer

2023-01-19 Thread GitBox
joarobles opened a new issue, #29036: URL: https://github.com/apache/airflow/issues/29036 ### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers _No response_ ### Apache Airflow version 2.3.3 ### Operating System MacOs/L

[GitHub] [airflow] potiuk merged pull request #29030: Fix `project-name` generation for Providers `test-type` in Breeze

2023-01-19 Thread GitBox
potiuk merged PR #29030: URL: https://github.com/apache/airflow/pull/29030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] snjypl commented on a diff in pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-19 Thread GitBox
snjypl commented on code in PR #28161: URL: https://github.com/apache/airflow/pull/28161#discussion_r1081083607 ## airflow/executors/kubernetes_executor.py: ## @@ -773,6 +775,57 @@ def _change_state(self, key: TaskInstanceKey, state: str | None, pod_id: str, na # d

[GitHub] [airflow] snjypl commented on a diff in pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-19 Thread GitBox
snjypl commented on code in PR #28161: URL: https://github.com/apache/airflow/pull/28161#discussion_r1081083607 ## airflow/executors/kubernetes_executor.py: ## @@ -773,6 +775,57 @@ def _change_state(self, key: TaskInstanceKey, state: str | None, pod_id: str, na # d

[GitHub] [airflow] eladkal commented on a diff in pull request #23079: Don't clear downstream tasks when marked status is failure and downstream is True

2023-01-19 Thread GitBox
eladkal commented on code in PR #23079: URL: https://github.com/apache/airflow/pull/23079#discussion_r1081025444 ## tests/models/test_dag.py: ## @@ -2925,6 +2926,57 @@ def consumer(value): ] +@pytest.mark.parametrize("run_id, execution_date", [(None, datetime_tz(2020,

[GitHub] [airflow] Taragolis opened a new pull request, #29035: Renaming nose compatible methods in flavour of regular pytest naming

2023-01-19 Thread GitBox
Taragolis opened a new pull request, #29035: URL: https://github.com/apache/airflow/pull/29035 In pytest 7.2 execute nose compatible methods such as `setup()`/`teardown()` was removed: https://github.com/pytest-dev/pytest/issues/9886 Even though we use old version of pytest it better

[GitHub] [airflow] rkarish commented on issue #27285: Using .output on non-templated fields

2023-01-19 Thread GitBox
rkarish commented on issue #27285: URL: https://github.com/apache/airflow/issues/27285#issuecomment-1396641848 Hey Jarek, I've opened a PR for this. I took a slightly different approach than a DAG level setting. I managed the functionality at the Operator level, but this can still be applie

[GitHub] [airflow] rkarish opened a new pull request, #29034: Add functionality for operators to template all eligible fields (apac…

2023-01-19 Thread GitBox
rkarish opened a new pull request, #29034: URL: https://github.com/apache/airflow/pull/29034 Closes: apache#27285 Related: apache#26938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ecerulm commented on a diff in pull request #29016: Fix leak sensitive field via V1EnvVar on exception

2023-01-19 Thread GitBox
ecerulm commented on code in PR #29016: URL: https://github.com/apache/airflow/pull/29016#discussion_r1080960124 ## airflow/utils/log/secrets_masker.py: ## @@ -200,10 +222,18 @@ def _redact(self, item: Redactable, name: str | None, depth: int) -> Redacted: if name

[GitHub] [airflow] BasPH commented on a diff in pull request #29027: Annotate and simplify code samples in DAGs doc

2023-01-19 Thread GitBox
BasPH commented on code in PR #29027: URL: https://github.com/apache/airflow/pull/29027#discussion_r1080956076 ## docs/apache-airflow/core-concepts/dags.rst: ## @@ -487,39 +522,50 @@ Unlike :ref:`concepts:subdags`, TaskGroups are purely a UI grouping concept. Tas .. image::

[GitHub] [airflow] phanikumv commented on a diff in pull request #29014: Add deferrable mode to `DbtCloudRunJobOperator`

2023-01-19 Thread GitBox
phanikumv commented on code in PR #29014: URL: https://github.com/apache/airflow/pull/29014#discussion_r1080939010 ## airflow/providers/dbt/cloud/hooks/dbt.py: ## @@ -125,6 +128,34 @@ class DbtCloudJobRunException(AirflowException): """An exception that indicates a job run

[GitHub] [airflow] phanikumv commented on a diff in pull request #29014: Add deferrable mode to `DbtCloudRunJobOperator`

2023-01-19 Thread GitBox
phanikumv commented on code in PR #29014: URL: https://github.com/apache/airflow/pull/29014#discussion_r1080937119 ## airflow/providers/dbt/cloud/operators/dbt.py: ## @@ -91,8 +96,17 @@ def __init__( timeout: int = 60 * 60 * 24 * 7, check_interval: int = 60,

[GitHub] [airflow] uranusjr commented on a diff in pull request #29016: Fix leak sensitive field via V1EnvVar on exception

2023-01-18 Thread GitBox
uranusjr commented on code in PR #29016: URL: https://github.com/apache/airflow/pull/29016#discussion_r1080905581 ## airflow/utils/log/secrets_masker.py: ## @@ -200,10 +222,18 @@ def _redact(self, item: Redactable, name: str | None, depth: int) -> Redacted: if name

[GitHub] [airflow] uranusjr commented on a diff in pull request #28942: Fix @task.kubernetes to receive input and send output

2023-01-18 Thread GitBox
uranusjr commented on code in PR #28942: URL: https://github.com/apache/airflow/pull/28942#discussion_r1080879736 ## airflow/providers/cncf/kubernetes/decorators/kubernetes.py: ## @@ -82,11 +83,42 @@ def _get_python_source(self): res = remove_task_decorator(res, "@task.

[GitHub] [airflow] uranusjr commented on a diff in pull request #27776: Add deferrable mode to dataflow operators

2023-01-18 Thread GitBox
uranusjr commented on code in PR #27776: URL: https://github.com/apache/airflow/pull/27776#discussion_r1080876447 ## airflow/providers/google/cloud/hooks/dataflow.py: ## @@ -719,6 +698,40 @@ def start_template_dataflow( jobs_controller.wait_for_done() return re

[GitHub] [airflow] uranusjr commented on a diff in pull request #27776: Add deferrable mode to dataflow operators

2023-01-18 Thread GitBox
uranusjr commented on code in PR #27776: URL: https://github.com/apache/airflow/pull/27776#discussion_r1080875713 ## airflow/providers/google/cloud/links/dataflow.py: ## @@ -48,5 +48,5 @@ def persist( operator_instance.xcom_push( context, key=D

[GitHub] [airflow] uranusjr commented on a diff in pull request #27776: Add deferrable mode to dataflow operators

2023-01-18 Thread GitBox
uranusjr commented on code in PR #27776: URL: https://github.com/apache/airflow/pull/27776#discussion_r1080875455 ## airflow/providers/google/cloud/hooks/dataflow.py: ## @@ -1037,7 +1050,7 @@ def start_sql_job( def get_job( self, job_id: str, -proj

[GitHub] [airflow] uranusjr commented on a diff in pull request #27776: Add deferrable mode to dataflow operators

2023-01-18 Thread GitBox
uranusjr commented on code in PR #27776: URL: https://github.com/apache/airflow/pull/27776#discussion_r1080874876 ## airflow/providers/google/cloud/hooks/dataflow.py: ## @@ -719,6 +698,40 @@ def start_template_dataflow( jobs_controller.wait_for_done() return re

[GitHub] [airflow] uranusjr commented on a diff in pull request #28894: Strip markup from `appbuilder.app_name` if `instance_name_has_markup = True`

2023-01-18 Thread GitBox
uranusjr commented on code in PR #28894: URL: https://github.com/apache/airflow/pull/28894#discussion_r1080869367 ## airflow/www/app.py: ## @@ -74,17 +76,37 @@ def sync_appbuilder_roles(flask_app): flask_app.appbuilder.sm.sync_roles() +def strip_tags(input_html: str

[GitHub] [airflow] Aakcht opened a new pull request, #29032: Chart: add doc note about podtemplate images

2023-01-18 Thread GitBox
Aakcht opened a new pull request, #29032: URL: https://github.com/apache/airflow/pull/29032 Add note that `images.pod_template.*` parameters in chart are overriden by `config.kubernetes` section. --- **^ Add meaningful description above** Read the **[Pull Request Gu

[GitHub] [airflow] uranusjr commented on issue #28870: Support multiple buttons from `operator_extra_links`

2023-01-18 Thread GitBox
uranusjr commented on issue #28870: URL: https://github.com/apache/airflow/issues/28870#issuecomment-1396486390 Using property is not supported because internally Airflow expects an iterable. Although this gives me an idea, we may be able to detect `operator_extra_links` is a property and h

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29031: Scheduler pod restarts

2023-01-18 Thread GitBox
boring-cyborg[bot] commented on issue #29031: URL: https://github.com/apache/airflow/issues/29031#issuecomment-1396472722 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] Subhashini2610 opened a new issue, #29031: Scheduler pod restarts

2023-01-18 Thread GitBox
Subhashini2610 opened a new issue, #29031: URL: https://github.com/apache/airflow/issues/29031 ### Apache Airflow version 2.5.0 ### What happened I have an instance of Airflow running on a K8s cluster. For some strange reason, the scheduler pods are restarted every now a

[GitHub] [airflow] uranusjr commented on a diff in pull request #29027: Annotate and simplify code samples in DAGs doc

2023-01-18 Thread GitBox
uranusjr commented on code in PR #29027: URL: https://github.com/apache/airflow/pull/29027#discussion_r1080820155 ## docs/apache-airflow/core-concepts/dags.rst: ## @@ -487,39 +522,50 @@ Unlike :ref:`concepts:subdags`, TaskGroups are purely a UI grouping concept. Tas .. image

[GitHub] [airflow] uranusjr merged pull request #29029: Add norm to triage group

2023-01-18 Thread GitBox
uranusjr merged PR #29029: URL: https://github.com/apache/airflow/pull/29029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
o-nikolas commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080786437 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,127 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor li

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28706: Update provide_bucket_name() decorator to handle new conn_type

2023-01-18 Thread GitBox
o-nikolas commented on code in PR #28706: URL: https://github.com/apache/airflow/pull/28706#discussion_r1080784133 ## docs/apache-airflow-providers-amazon/connections/aws.rst: ## @@ -67,6 +67,8 @@ Extra (optional) Specify the extra parameters (as json dictionary) that can b

[GitHub] [airflow] o-nikolas merged pull request #28925: Add log for AWS Glue Job Console URL

2023-01-18 Thread GitBox
o-nikolas merged PR #28925: URL: https://github.com/apache/airflow/pull/28925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] o-nikolas merged pull request #29028: Patch MockExecutor in tests instead of "prod" code

2023-01-18 Thread GitBox
o-nikolas merged PR #29028: URL: https://github.com/apache/airflow/pull/29028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] j-svensmark commented on issue #29002: KubernetesPodOperator xcom push failure

2023-01-18 Thread GitBox
j-svensmark commented on issue #29002: URL: https://github.com/apache/airflow/issues/29002#issuecomment-1396358849 Sure, I'm seeing the issue with the following DAG ``` import airflow import os import pendulum from airflow import DAG from airflow.providers.cncf.kubernete

[GitHub] [airflow] josh-fell opened a new pull request, #29030: Fix `project-name` generation for Providers `test-type` in Breeze

2023-01-18 Thread GitBox
josh-fell opened a new pull request, #29030: URL: https://github.com/apache/airflow/pull/29030 The `breeze testing tests --test-type "Providers[...]"` command would fail for an invalid `project-name` for `docker compose` if the provider(s) specified required a dotted path (e.g. `microsoft.a

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28686: Update gantt chart UI to display queued state of tasks

2023-01-18 Thread GitBox
boring-cyborg[bot] commented on PR #28686: URL: https://github.com/apache/airflow/pull/28686#issuecomment-1396304738 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] bbovenzi merged pull request #28686: Update gantt chart UI to display queued state of tasks

2023-01-18 Thread GitBox
bbovenzi merged PR #28686: URL: https://github.com/apache/airflow/pull/28686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28932: Update how PythonSensor returns values from python_callable

2023-01-18 Thread GitBox
boring-cyborg[bot] commented on PR #28932: URL: https://github.com/apache/airflow/pull/28932#issuecomment-1396304153 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] Taragolis merged pull request #28932: Update how PythonSensor returns values from python_callable

2023-01-18 Thread GitBox
Taragolis merged PR #28932: URL: https://github.com/apache/airflow/pull/28932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] jedcunningham opened a new pull request, #29029: Add norm to triage group

2023-01-18 Thread GitBox
jedcunningham opened a new pull request, #29029: URL: https://github.com/apache/airflow/pull/29029 @norm is going to help manage issues for AIP-52. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] github-actions[bot] closed pull request #26004: [Loki log handler] - Integration with Grafana Loki

2023-01-18 Thread GitBox
github-actions[bot] closed pull request #26004: [Loki log handler] - Integration with Grafana Loki URL: https://github.com/apache/airflow/pull/26004 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] github-actions[bot] closed pull request #26269: Added Feature - Glue job continuously log printing in Airflow task logs

2023-01-18 Thread GitBox
github-actions[bot] closed pull request #26269: Added Feature - Glue job continuously log printing in Airflow task logs URL: https://github.com/apache/airflow/pull/26269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [airflow] github-actions[bot] closed pull request #27787: POC / WIP testing out using logfmt for more structured logging

2023-01-18 Thread GitBox
github-actions[bot] closed pull request #27787: POC / WIP testing out using logfmt for more structured logging URL: https://github.com/apache/airflow/pull/27787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [airflow] github-actions[bot] commented on pull request #27247: Added extra annotations support

2023-01-18 Thread GitBox
github-actions[bot] commented on PR #27247: URL: https://github.com/apache/airflow/pull/27247#issuecomment-1396265686 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] Taragolis commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
Taragolis commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396227391 Previously we run `LocalTaskJob` in the same process as we run pytest, as result we had unstable result when we tried to terminate this process `LocalTaskJob` prevent but something ac

[GitHub] [airflow] potiuk commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
potiuk commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396209676 > Well, it seems not always we could grab logs from stdout. E.. I :heart: flaky tests :smiling_face_with_tear: -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] eladkal commented on a diff in pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-18 Thread GitBox
eladkal commented on code in PR #28161: URL: https://github.com/apache/airflow/pull/28161#discussion_r1080630500 ## airflow/executors/kubernetes_executor.py: ## @@ -773,6 +775,57 @@ def _change_state(self, key: TaskInstanceKey, state: str | None, pod_id: str, na #

[GitHub] [airflow] Taragolis commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
Taragolis commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396179586 Well, seems not always we could grab logs from stdout as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] o-nikolas opened a new pull request, #29028: Patch MockExecutor in tests instead of "prod" code

2023-01-18 Thread GitBox
o-nikolas opened a new pull request, #29028: URL: https://github.com/apache/airflow/pull/29028 tl;dr: Patch the MockExecutor into the executor list within test code rather than core/prod code. Full story: The MockExecutor, which is used for testing, was added to the list of executors

[GitHub] [airflow] potiuk merged pull request #29023: Update owner list for release dockerhub image workflow

2023-01-18 Thread GitBox
potiuk merged PR #29023: URL: https://github.com/apache/airflow/pull/29023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] kolfild26 commented on a diff in pull request #28998: write to DB a user who triggered a dag via API endpoint request

2023-01-18 Thread GitBox
kolfild26 commented on code in PR #28998: URL: https://github.com/apache/airflow/pull/28998#discussion_r1080595774 ## airflow/api_connexion/endpoints/dag_run_endpoint.py: ## @@ -319,6 +320,27 @@ def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:

[GitHub] [airflow] vincbeck commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
vincbeck commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080590765 ## airflow/providers/amazon/aws/example_dags/example_s3_to_sql.py: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment:

[GitHub] [airflow] vincbeck commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
vincbeck commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080589748 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor lic

[GitHub] [airflow] vincbeck commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
vincbeck commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080580457 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,127 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor lic

[GitHub] [airflow] Taragolis commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
Taragolis commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396115730 All checks have passed 🥳 🔥 And I think I found the way how to check logs records again. Anyway to early to celebrate, need much more success runs. -- This is an automated mes

[GitHub] [airflow] pierrejeambrun commented on issue #29026: Status of testing of Apache Airflow 2.5.1rc2

2023-01-18 Thread GitBox
pierrejeambrun commented on issue #29026: URL: https://github.com/apache/airflow/issues/29026#issuecomment-1396109636 Total fine by me, I wondered what to do about that, thanks Jarek :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] hussein-awala commented on issue #29018: Able to pass arguments to `--conf` not specified in `parameters` argument

2023-01-18 Thread GitBox
hussein-awala commented on issue #29018: URL: https://github.com/apache/airflow/issues/29018#issuecomment-1396107617 The problem is that there is currently no way to provide `params` directly, instead we provide the parameters as `dag_run.conf` which is a free dictionary without any validat

[GitHub] [airflow] BasPH opened a new pull request, #29027: Annotate and simplify code samples in DAGs doc

2023-01-18 Thread GitBox
BasPH opened a new pull request, #29027: URL: https://github.com/apache/airflow/pull/29027 This PR updates several code samples in https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html: - Added annotations wherever necessary - Fix indentation - Added imports i

[GitHub] [airflow] maggesssss commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
magges commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080570241 ## airflow/providers/amazon/aws/example_dags/example_s3_to_sql.py: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Commen

[GitHub] [airflow] potiuk commented on issue #29026: Status of testing of Apache Airflow 2.5.1rc2

2023-01-18 Thread GitBox
potiuk commented on issue #29026: URL: https://github.com/apache/airflow/issues/29026#issuecomment-1396104668 I allowed myself to copy the "checked" status from #28947 - > I guess the changes were small enough to assume what's been tested is still working fine :) -- This is an automated m

[GitHub] [airflow] maggesssss commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-18 Thread GitBox
magges commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1080568360 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor l

[GitHub] [airflow] pierrejeambrun closed issue #28947: Status of testing of Apache Airflow 2.5.1rc1

2023-01-18 Thread GitBox
pierrejeambrun closed issue #28947: Status of testing of Apache Airflow 2.5.1rc1 URL: https://github.com/apache/airflow/issues/28947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [airflow] pierrejeambrun commented on issue #28947: Status of testing of Apache Airflow 2.5.1rc1

2023-01-18 Thread GitBox
pierrejeambrun commented on issue #28947: URL: https://github.com/apache/airflow/issues/28947#issuecomment-1396095716 Thanks to all that helped testing rc1. An rc2 has been issued, closing in favor of https://github.com/apache/airflow/issues/29026 -- This is an automated message from the

[GitHub] [airflow] pierrejeambrun opened a new issue, #29026: Status of testing of Apache Airflow 2.5.1rc2

2023-01-18 Thread GitBox
pierrejeambrun opened a new issue, #29026: URL: https://github.com/apache/airflow/issues/29026 ### Body We have a kind request for all the contributors to the latest [Apache Airflow RC 2.5.1rc2](https://pypi.org/project/apache-airflow/2.5.1rc2/). Could you please help us to tes

[GitHub] [airflow] potiuk merged pull request #29024: Clarify that we are using lowest supported version for static checks

2023-01-18 Thread GitBox
potiuk merged PR #29024: URL: https://github.com/apache/airflow/pull/29024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk merged pull request #28996: Skip DockerOperator task when it returns a provided exit code

2023-01-18 Thread GitBox
potiuk merged PR #28996: URL: https://github.com/apache/airflow/pull/28996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk closed issue #28951: Add a way to skip Docker Operator task

2023-01-18 Thread GitBox
potiuk closed issue #28951: Add a way to skip Docker Operator task URL: https://github.com/apache/airflow/issues/28951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[GitHub] [airflow] Taragolis commented on a diff in pull request #28925: Add log for AWS Glue Job Console URL

2023-01-18 Thread GitBox
Taragolis commented on code in PR #28925: URL: https://github.com/apache/airflow/pull/28925#discussion_r1080548545 ## airflow/providers/amazon/aws/operators/glue.py: ## @@ -144,6 +149,20 @@ def execute(self, context: Context): self.wait_for_completion, )

[GitHub] [airflow] snjypl commented on pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-18 Thread GitBox
snjypl commented on PR #28161: URL: https://github.com/apache/airflow/pull/28161#issuecomment-1396069446 @potiuk @eladkal @pierrejeambrun will be great if you could review this PR whenever you get a chance ! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] potiuk commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
potiuk commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1396065868 :crossed_fingers: :crossed_fingers: :crossed_fingers: :crossed_fingers: :crossed_fingers: :crossed_fingers: -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] Taragolis commented on a diff in pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-18 Thread GitBox
Taragolis commented on code in PR #29001: URL: https://github.com/apache/airflow/pull/29001#discussion_r1080537555 ## airflow/providers/amazon/aws/sensors/glue.py: ## @@ -61,10 +62,13 @@ def __init__( self.errored_states: list[str] = ["FAILED", "STOPPED", "TIMEOUT"]

[GitHub] [airflow] collinmcnulty opened a new issue, #29025: Allow K8s Executor to add logs for tasks that failed to launch as a worker pod

2023-01-18 Thread GitBox
collinmcnulty opened a new issue, #29025: URL: https://github.com/apache/airflow/issues/29025 ### Description The Kube Watcher in the K8s executor keeps track of tasks that are intended to be launched as K8s pods. If something happens to these pods, such as they fail to launch before

[GitHub] [airflow] potiuk opened a new pull request, #29024: Clarify that we are using lowest supported version for static checks

2023-01-18 Thread GitBox
potiuk opened a new pull request, #29024: URL: https://github.com/apache/airflow/pull/29024 There is a good reason static checks always use lowest supported Python version. This PR clarifies that it happens and why. --- **^ Add meaningful description above** Read the

[GitHub] [airflow] travis-cook-sfdc commented on issue #28870: Support multiple buttons from `operator_extra_links`

2023-01-18 Thread GitBox
travis-cook-sfdc commented on issue #28870: URL: https://github.com/apache/airflow/issues/28870#issuecomment-1387745349 > nice discussion. I see the problem now, and yeah, I agree having support for dynamic collection of links in operator is a better idea @potiuk - It does seem like `

[GitHub] [airflow] pierrejeambrun opened a new pull request, #29023: Update owner list for release dockerhub image workflow

2023-01-18 Thread GitBox
pierrejeambrun opened a new pull request, #29023: URL: https://github.com/apache/airflow/pull/29023 Add pierrejeambrun to owners of `Release PROD images` workflow. Tried to keep the list sorted :) -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-18 Thread GitBox
hussein-awala commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1074043280 ## airflow/utils/json.py: ## @@ -59,7 +59,11 @@ def default(self, o: Any) -> Any: data = serialize(o) if isinstance(data, dict) and DA

[GitHub] [airflow] travis-cook-sfdc commented on issue #28870: Support multiple buttons from `operator_extra_links`

2023-01-18 Thread GitBox
travis-cook-sfdc commented on issue #28870: URL: https://github.com/apache/airflow/issues/28870#issuecomment-1387738430 After a little more investigation, I think that example only works because the extra link is registered as a provider which doesn't require explicit instantiation. If you

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-18 Thread GitBox
hussein-awala commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1074037887 ## airflow/utils/json.py: ## @@ -71,7 +75,7 @@ def default(self, o: Any) -> Any: return data[DATA] return data except

[GitHub] [airflow] Taragolis commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`

2023-01-18 Thread GitBox
Taragolis commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1387730994 Ok, lets try to next hack - run this test in separate process. After this changes it even work fine in MacOS, lets have a look is it pass even single run in the CI -- This is an

[GitHub] [airflow] BasPH merged pull request #29021: [docs] change context example to no longer use as variable

2023-01-18 Thread GitBox
BasPH merged PR #29021: URL: https://github.com/apache/airflow/pull/29021 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.ap

[GitHub] [airflow] vchiapaikeo commented on issue #29002: KubernetesPodOperator xcom push failure

2023-01-18 Thread GitBox
vchiapaikeo commented on issue #29002: URL: https://github.com/apache/airflow/issues/29002#issuecomment-1387719290 Fwiw, I'm on 2.5.0 with cncf 5.0.0 and cannot repro this on our installation. @j-svensmark , can you share a minimal example dag that shows this issue? -- This is an automat

[GitHub] [airflow] potiuk opened a new pull request, #29022: Only skip provider integration tests for non-main builds

2023-01-18 Thread GitBox
potiuk opened a new pull request, #29022: URL: https://github.com/apache/airflow/pull/29022 Previously we skipped all integration tests in non-main builds (i.e. builds that were running in v2-* branches where we test and prepare releases. This had two effects: * the non-provider inte

[GitHub] [airflow] potiuk commented on pull request #28569: Add general-purpose "notifier" concept to DAGs

2023-01-18 Thread GitBox
potiuk commented on PR #28569: URL: https://github.com/apache/airflow/pull/28569#issuecomment-1387709942 > > Should we add docs for this feature @ephraimbuddy ? Currently this feature is a "hidden" one > > Seeing this now. There's a doc on it: http://apache-airflow-docs.s3-website.eu

[GitHub] [airflow] ephraimbuddy closed issue #28071: Kubernetes logging errors - attempting to adopt taskinstance which was not specified by database

2023-01-18 Thread GitBox
ephraimbuddy closed issue #28071: Kubernetes logging errors - attempting to adopt taskinstance which was not specified by database URL: https://github.com/apache/airflow/issues/28071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] ephraimbuddy merged pull request #28899: Be more selective when adopting pods with KubernetesExecutor

2023-01-18 Thread GitBox
ephraimbuddy merged PR #28899: URL: https://github.com/apache/airflow/pull/28899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[GitHub] [airflow] ferruzzi commented on a diff in pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-18 Thread GitBox
ferruzzi commented on code in PR #29001: URL: https://github.com/apache/airflow/pull/29001#discussion_r1073947810 ## airflow/providers/amazon/aws/sensors/glue.py: ## @@ -61,10 +62,13 @@ def __init__( self.errored_states: list[str] = ["FAILED", "STOPPED", "TIMEOUT"]

[GitHub] [airflow] travis-cook-sfdc commented on issue #28870: Support multiple buttons from `operator_extra_links`

2023-01-18 Thread GitBox
travis-cook-sfdc commented on issue #28870: URL: https://github.com/apache/airflow/issues/28870#issuecomment-1387616659 @Taragolis - Interesting example. I thought I had tried something similar, but I ran into problems when I attempted to register subclass of `BaseOperatorLink` as an Airflo

[GitHub] [airflow] ferruzzi commented on a diff in pull request #29001: uniformize getting hook through cached property in aws sensors

2023-01-18 Thread GitBox
ferruzzi commented on code in PR #29001: URL: https://github.com/apache/airflow/pull/29001#discussion_r1073947810 ## airflow/providers/amazon/aws/sensors/glue.py: ## @@ -61,10 +62,13 @@ def __init__( self.errored_states: list[str] = ["FAILED", "STOPPED", "TIMEOUT"]

[GitHub] [airflow] ephraimbuddy commented on pull request #28569: Add general-purpose "notifier" concept to DAGs

2023-01-18 Thread GitBox
ephraimbuddy commented on PR #28569: URL: https://github.com/apache/airflow/pull/28569#issuecomment-1387521950 > Should we add docs for this feature @ephraimbuddy ? Currently this feature is a "hidden" one Seeing this now. There's a doc on it: http://apache-airflow-docs.s3-website.eu

[GitHub] [airflow] potiuk commented on issue #11463: Tags rather than names in variable parts of the metrics

2023-01-18 Thread GitBox
potiuk commented on issue #11463: URL: https://github.com/apache/airflow/issues/11463#issuecomment-1387512719 Yeah. Given the context (thanks for that) I think small - non-breaking especially - improvements in this area are still good tactical solution even if the long-term strategic approa

<    1   2   3   4   5   6   7   8   9   10   >