[GitHub] [airflow] phanikumv commented on a diff in pull request #26345: Add DataFlow operations to Azure DataFactory hook

2022-09-14 Thread GitBox
phanikumv commented on code in PR #26345: URL: https://github.com/apache/airflow/pull/26345#discussion_r971030906 ## airflow/providers/microsoft/azure/hooks/data_factory.py: ## @@ -479,12 +481,121 @@ def delete_dataset( Delete the dataset. :param

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #26411: update contributing documentations

2022-09-14 Thread GitBox
boring-cyborg[bot] commented on PR #26411: URL: https://github.com/apache/airflow/pull/26411#issuecomment-1247490041 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] Lee-W opened a new pull request, #26411: update contributing documentations

2022-09-14 Thread GitBox
Lee-W opened a new pull request, #26411: URL: https://github.com/apache/airflow/pull/26411 * CONTRIBUTING.rst: replace markdown style link as rst style * CONTRIBUTORS_QUICK_START.rst * unify python version in examples to 3.8 (originally, the first example uses 3.7 which could

[airflow] branch v2-4-test updated (a8ce50986d -> 7ba5963756)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit a8ce50986d Update Release Note add 3871f00230 Fix UI redirect (#26409) add c7ea01d676 Better

[GitHub] [airflow] wookiist commented on issue #23145: Task stuck in "scheduled" when running in backfill job

2022-09-14 Thread GitBox
wookiist commented on issue #23145: URL: https://github.com/apache/airflow/issues/23145#issuecomment-1247451952 Is there any update regarding this issue? There are many tasks that fall into the 'scheduled' state when working on the backfill  Apache Airflow 2.3.4 / Kubernetes

[airflow] branch main updated: Adding podAnnotations to StatsD deployment template (#25732)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 951b708461 Adding podAnnotations to StatsD

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #25732: Adding podAnnotations to StatsD deployment template

2022-09-14 Thread GitBox
boring-cyborg[bot] commented on PR #25732: URL: https://github.com/apache/airflow/pull/25732#issuecomment-1247434280 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] jedcunningham closed issue #25446: Helm Chart: Allow adding annotations to statsd deployment

2022-09-14 Thread GitBox
jedcunningham closed issue #25446: Helm Chart: Allow adding annotations to statsd deployment URL: https://github.com/apache/airflow/issues/25446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] jedcunningham merged pull request #25732: Adding podAnnotations to StatsD deployment template

2022-09-14 Thread GitBox
jedcunningham merged PR #25732: URL: https://github.com/apache/airflow/pull/25732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated (56e7555c42 -> fdecf12051)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 56e7555c42 Fix UI redirect (#26409) add fdecf12051 Add min attrs version (#26408) No new revisions were

[GitHub] [airflow] jedcunningham merged pull request #26408: Add min attrs version

2022-09-14 Thread GitBox
jedcunningham merged PR #26408: URL: https://github.com/apache/airflow/pull/26408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: Fix UI redirect (#26409)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 56e7555c42 Fix UI redirect (#26409)

[GitHub] [airflow] jedcunningham merged pull request #26409: Fix UI redirect

2022-09-14 Thread GitBox
jedcunningham merged PR #26409: URL: https://github.com/apache/airflow/pull/26409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham closed issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
jedcunningham closed issue #26405: helm chart fails during `post-install` URL: https://github.com/apache/airflow/issues/26405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] eeluve commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
eeluve commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247426380 ❤️ thanks man solved also, a note: I'll quite likely kill my colleague, that I pinged 5 hours ago about whether this cluster is fine lets just think of this as some nice

[GitHub] [airflow] jedcunningham commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
jedcunningham commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247419205 Sorry, I went from memory and got it wrong, it'd be `postgresql.primary.nodeSelector`! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] eeluve commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
eeluve commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247418616 thanks, checking `postgres.primary.nodeSelector`. I'll update this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] jedcunningham commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
jedcunningham commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247418011 Something cluster related seems likely. When you describe any of the pending pods, normally you'll see why in the events. In the OP I had a bit of trouble determining if it

[GitHub] [airflow] eeluve commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
eeluve commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247413155 it still doesn't work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] jedcunningham commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
jedcunningham commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247407598 Does this work? ``` helm upgrade --install airflow apache-airflow/airflow ``` If so, slowly add stuff in until it doesn't. That might help us narrow down

[GitHub] [airflow] jedcunningham commented on a diff in pull request #26341: Improve 'start-airflow' experience for users and ui developers

2022-09-14 Thread GitBox
jedcunningham commented on code in PR #26341: URL: https://github.com/apache/airflow/pull/26341#discussion_r971373620 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -213,6 +213,18 @@ def shell( @option_mount_sources @option_integration

[GitHub] [airflow] ferruzzi opened a new pull request, #26410: Convert `example_eks_with_nodegroup_in_one_step` sample DAG to system test (AIP-47)

2022-09-14 Thread GitBox
ferruzzi opened a new pull request, #26410: URL: https://github.com/apache/airflow/pull/26410 Follows the template set forward in https://github.com/apache/airflow/pull/24643 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] jedcunningham commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0

2022-09-14 Thread GitBox
jedcunningham commented on code in PR #26371: URL: https://github.com/apache/airflow/pull/26371#discussion_r971365529 ## RELEASE_NOTES.rst: ## @@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08) Significant Changes ^^^ -- The DB related classes:

[GitHub] [airflow] dstandish commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dstandish commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-124734 @dimberman got this deprecation warning which ... probably you can avoid somehow: ``` local ❯ airflow dags test example_time_delta_sensor_async -S

[airflow] branch main updated (01e3fb7eac -> e458eb689d)

2022-09-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 01e3fb7eac Clear autoregistered DAGs if there are any import errors (#26398) add e458eb689d Fix params reference

[GitHub] [airflow] potiuk merged pull request #26406: Fix params reference for default_constraints_branch

2022-09-14 Thread GitBox
potiuk merged PR #26406: URL: https://github.com/apache/airflow/pull/26406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated (f6c579c1c0 -> 01e3fb7eac)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f6c579c1c0 Require dag_id arg for dags list-runs (#26357) add 01e3fb7eac Clear autoregistered DAGs if

[GitHub] [airflow] jedcunningham merged pull request #26398: Clear autoregistered DAGs if there are any import errors

2022-09-14 Thread GitBox
jedcunningham merged PR #26398: URL: https://github.com/apache/airflow/pull/26398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247337394 > @potiuk could we make the debugger a separate discussion? I don't personally have a strong opinion one way or another, but it sounds like that might be something that could slow down

[GitHub] [airflow] jedcunningham opened a new pull request, #26408: Add min attrs version

2022-09-14 Thread GitBox
jedcunningham opened a new pull request, #26408: URL: https://github.com/apache/airflow/pull/26408 I'm not sure when we started using attrs directly, but we do and we need >=22.1.0 as we use the min_length validator. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247330258 (alledged fix that is - I tested it locally and it worked to create the k8s venv ;) - but you never know in CI). -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247329595 Fix in #26407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk commented on pull request #26407: Attempt to create k8s venv without constraints if constraint build fails

2022-09-14 Thread GitBox
potiuk commented on PR #26407: URL: https://github.com/apache/airflow/pull/26407#issuecomment-1247329222 This one is needed in order to make changes like #26393 to pass the k8s tests. This change adds FAB version bump - so that they are conflicting with the current constraints. And this

[GitHub] [airflow] potiuk opened a new pull request, #26407: Attempt to create k8s venv without constraints if constraint build fails

2022-09-14 Thread GitBox
potiuk opened a new pull request, #26407: URL: https://github.com/apache/airflow/pull/26407 There is an issue with trying to install the k8s env where the current requirements conflict with constraints. The k8s env creation is failing in this case. This PR will not fail hard when

[GitHub] [airflow] gmsantos commented on pull request #26406: Fix params reference for default_constraints_branch

2022-09-14 Thread GitBox
gmsantos commented on PR #26406: URL: https://github.com/apache/airflow/pull/26406#issuecomment-1247326244 ``` Moved Permanently. ``` curl is getting this message because it's trying to download the following file pattern:

[GitHub] [airflow] potiuk commented on pull request #26406: Fix params reference for default_constraints_branch

2022-09-14 Thread GitBox
potiuk commented on PR #26406: URL: https://github.com/apache/airflow/pull/26406#issuecomment-1247325586 nice one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] gmsantos opened a new pull request, #26406: Fix params reference for default_constraints_branch

2022-09-14 Thread GitBox
gmsantos opened a new pull request, #26406: URL: https://github.com/apache/airflow/pull/26406 Fixes breeze commands that depends on the default constraints branch, such as the generated diff for `breeze release-management generate-constraints` -- This is an automated message

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247315599 > Unfortunately, this PR didn't pass the [CI step check](https://github.com/apache/airflow/blob/main/.github/workflows/ci.yml#L1635) for a canary run: Told ya :). We are both

[airflow] branch v2-4-test updated (6bda3ae243 -> a8ce50986d)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit 6bda3ae243 Update Release Note add a8ce50986d Update Release Note This update added new revisions

[GitHub] [airflow] blag commented on pull request #26376: Add dataset event timestamp to dataset dag run queue

2022-09-14 Thread GitBox
blag commented on PR #26376: URL: https://github.com/apache/airflow/pull/26376#issuecomment-1247301953 Ease of testing is a perfectly valid and good reason. You've convinced me.  And yeah, I was thinking the dataset event would be the last of all of the dependencies that was updated

[GitHub] [airflow] eeluve opened a new issue, #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
eeluve opened a new issue, #26405: URL: https://github.com/apache/airflow/issues/26405 ### Official Helm Chart version 1.6.0 (latest released) ### Apache Airflow version 2.3.0 ### Kubernetes Version v1.20.6 ### Helm Chart configuration - helm

[GitHub] [airflow] boring-cyborg[bot] commented on issue #26405: helm chart fails during `post-install`

2022-09-14 Thread GitBox
boring-cyborg[bot] commented on issue #26405: URL: https://github.com/apache/airflow/issues/26405#issuecomment-1247290943 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] dimberman commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971284109 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] gmsantos commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
gmsantos commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247288007 Unfortunately, this PR didn't pass the [CI step check](https://github.com/apache/airflow/blob/main/.github/workflows/ci.yml#L1635) for a canary run:

[airflow] branch v2-4-test updated (ec9552d53d -> 6bda3ae243)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit ec9552d53d Update Release Note add 6bda3ae243 Update Release Note This update added new revisions

[airflow] branch v2-4-test updated (98af5a6c54 -> ec9552d53d)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit 98af5a6c54 Apply suggestions from code review omit 4a4dc40068 Update Release Note add

[GitHub] [airflow] dimberman commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247279974 @potiuk could we make the debugger a separate discussion? I don't personally have a strong opinion one way or another, but it sounds like that might be something that could slow down

[GitHub] [airflow] vincbeck commented on a diff in pull request #26221: Remove log group from AWS batch system test

2022-09-14 Thread GitBox
vincbeck commented on code in PR #26221: URL: https://github.com/apache/airflow/pull/26221#discussion_r971273303 ## tests/system/providers/amazon/aws/example_batch.py: ## @@ -15,6 +15,7 @@ # specific language governing permissions and limitations # under the License. from

[GitHub] [airflow] dstandish commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dstandish commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971272469 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] dstandish commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dstandish commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971272230 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] ashb commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971269263 ## airflow/models/dag.py: ## @@ -2435,6 +2436,51 @@ def cli(self): args = parser.parse_args() args.func(args, self) +def test(self,

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0

2022-09-14 Thread GitBox
ephraimbuddy commented on code in PR #26371: URL: https://github.com/apache/airflow/pull/26371#discussion_r971254574 ## RELEASE_NOTES.rst: ## @@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08) Significant Changes ^^^ -- The DB related classes:

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0

2022-09-14 Thread GitBox
ephraimbuddy commented on code in PR #26371: URL: https://github.com/apache/airflow/pull/26371#discussion_r971249686 ## RELEASE_NOTES.rst: ## @@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08) Significant Changes ^^^ -- The DB related classes:

[GitHub] [airflow] dimberman commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971244568 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] dstandish commented on pull request #26376: Add dataset event timestamp to dataset dag run queue

2022-09-14 Thread GitBox
dstandish commented on PR #26376: URL: https://github.com/apache/airflow/pull/26376#issuecomment-1247241432 > My personal preference would be to pass around the DatasetModel and DatasetEvent objects directly ultimately, they are still private, so, if someone wants to change it back,

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247240375 > Oh yes. Still, see my previous point about breakpoint() then breakpoint is fine. but it will not give you a fully-fledged, intuitive debugger in the console wiht visual

[GitHub] [airflow] dimberman commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971238703 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +523,60 @@ def dag_test(args, session=None): print(dot_graph.source) +@provide_session +def

[GitHub] [airflow] ashb commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971237931 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] ashb commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971237478 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +524,56 @@ def dag_test(args, session=None): print(dot_graph.source) +def _run_task(ti:

[GitHub] [airflow] ashb commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247232727 > We already have DebugExecutor that does exactly that in the same way and a bit simpler (and is portable across IDEs) I think..

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247225730 > Sneak peak > > 2022-09-14.20-34-13.webm We already have DebugExecutor that does exactly that in the same way and a bit simpler I think..

[GitHub] [airflow] ashb commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971228491 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +523,60 @@ def dag_test(args, session=None): print(dot_graph.source) +@provide_session +def

[GitHub] [airflow] potiuk commented on pull request #26341: Improve 'start-airflow' experience for users and ui developers

2022-09-14 Thread GitBox
potiuk commented on PR #26341: URL: https://github.com/apache/airflow/pull/26341#issuecomment-1247222862 Looking forward to merging this one :). Should be good improvement for many users. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk commented on pull request #26341: Improve 'start-airflow' experience for users and ui developers

2022-09-14 Thread GitBox
potiuk commented on PR #26341: URL: https://github.com/apache/airflow/pull/26341#issuecomment-1247223116 (rebased it to include PEP-563) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ashb commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247220745 Sneak peak [2022-09-14 20-34-13.webm](https://user-images.githubusercontent.com/34150/190246852-aabf03aa-3588-4fca-b652-e8c0a40d0b61.webm) -- This is an automated message

[GitHub] [airflow] dimberman commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971225589 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +523,60 @@ def dag_test(args, session=None): print(dot_graph.source) +@provide_session +def

[airflow] branch v2-4-test updated (4a4dc40068 -> 98af5a6c54)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git from 4a4dc40068 Update Release Note add 98af5a6c54 Apply suggestions from code review No new revisions

[GitHub] [airflow] ashb commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247206686 It needs a bit of work/polish, but using this ```json { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes.

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247195492 Or name it `--debug-pudb` as a flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247195130 We can even do not make it a requiremeent, but install it dynamically when someone uses `--debug` option BTW. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247192162 > Thank you @potiuk ! > > Do you think that would be too oppinionated to choose a specific debugger? One really awesome thing we could potentially do with this is even write a

[GitHub] [airflow] DMilmont opened a new pull request, #26401: Update manage-dags-files.rst

2022-09-14 Thread GitBox
DMilmont opened a new pull request, #26401: URL: https://github.com/apache/airflow/pull/26401 This removes the wording that baking DAGs into a docker image is the recommended way. Additionally it calls out the downside of doing so. closes:

[GitHub] [airflow] ashb commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247187636 I don't think we should integrate any specific debugger here in this CLI function, but instruct people to use `breakpoint()` and then either it "just works" with their IDE's integrated

[GitHub] [airflow] ashb commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
ashb commented on code in PR #26400: URL: https://github.com/apache/airflow/pull/26400#discussion_r971187508 ## airflow/cli/commands/dag_command.py: ## @@ -497,6 +523,60 @@ def dag_test(args, session=None): print(dot_graph.source) +@provide_session +def

[GitHub] [airflow] dimberman commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247181981 Thank you @potiuk ! Do you think that would be too oppinionated to choose a specific debugger? One really awesome thing we could potentially do with this is even write a guide

[GitHub] [airflow] blag commented on pull request #26376: Add dataset event timestamp to dataset dag run queue

2022-09-14 Thread GitBox
blag commented on PR #26376: URL: https://github.com/apache/airflow/pull/26376#issuecomment-1247151568 My personal preference would be to pass around the `DatasetModel` and `DatasetEvent` objects directly, since that just seems cleaner to me. If/when we publicize more/all of the dataset

[airflow] branch constraints-main updated: Updating constraints. Build id:3054654991

2022-09-14 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new de1a5e15c4 Updating

[GitHub] [airflow] potiuk commented on issue #13853: Clearing of historic Task or DagRuns leads to failed DagRun

2022-09-14 Thread GitBox
potiuk commented on issue #13853: URL: https://github.com/apache/airflow/issues/13853#issuecomment-1247133724 Thanks for confirming it ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247132314 > Nice workflow @potiuk ! 爛 When we see it works, yes. I am always sceptical about those kind of changes in CI, because the only way to test them is to try them live :). So I

[GitHub] [airflow] jedcunningham commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0

2022-09-14 Thread GitBox
jedcunningham commented on code in PR #26371: URL: https://github.com/apache/airflow/pull/26371#discussion_r971145746 ## RELEASE_NOTES.rst: ## @@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08) Significant Changes ^^^ -- The DB related classes:

[GitHub] [airflow] jedcunningham commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0

2022-09-14 Thread GitBox
jedcunningham commented on code in PR #26371: URL: https://github.com/apache/airflow/pull/26371#discussion_r971140772 ## RELEASE_NOTES.rst: ## @@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08) Significant Changes ^^^ -- The DB related classes:

[GitHub] [airflow] potiuk commented on pull request #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
potiuk commented on PR #26400: URL: https://github.com/apache/airflow/pull/26400#issuecomment-1247118543 This looks sper cool :). I will do some quick testing to see the experience. Few suggestions (asking more what you think about it and maybe those could be separate PRs., or

[GitHub] [airflow] oz-r commented on issue #13853: Clearing of historic Task or DagRuns leads to failed DagRun

2022-09-14 Thread GitBox
oz-r commented on issue #13853: URL: https://github.com/apache/airflow/issues/13853#issuecomment-1247111854 Finally got around to updating Airflow and it seems to be working as expected now. -- This is an automated message from the Apache Git Service. To respond to the message, please

[airflow] branch v2-4-test updated (71cca39c2d -> 4a4dc40068)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard 71cca39c2d Update release note discard fb89ce9dcf Work around pyupgrade edge cases (#26384) discard

[GitHub] [airflow] gmsantos commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
gmsantos commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247109591 Nice workflow @potiuk ! 爛 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] blag commented on pull request #26376: Add dataset event timestamp to dataset dag run queue

2022-09-14 Thread GitBox
blag commented on PR #26376: URL: https://github.com/apache/airflow/pull/26376#issuecomment-1247107735 Just to color in my thoughts on the private methods in the dataset manager - I made them private only because I was trying to keep the dataset manager public interface as small as

[GitHub] [airflow] potiuk commented on issue #26397: Deadlock dag_runs when max_active_runs=1 and dag_duration greater then schedule interval

2022-09-14 Thread GitBox
potiuk commented on issue #26397: URL: https://github.com/apache/airflow/issues/26397#issuecomment-1247105188 I think we had some issues fixed around that recently. Can you please upgrade to 2.3.4 and check it there (you will have to do it anyway and you can save us investigation and

[airflow] branch v2-4-test updated (fb89ce9dcf -> 71cca39c2d)

2022-09-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git from fb89ce9dcf Work around pyupgrade edge cases (#26384) add 71cca39c2d Update release note No new

[GitHub] [airflow] potiuk commented on a diff in pull request #26384: Work around pyupgrade edge cases

2022-09-14 Thread GitBox
potiuk commented on code in PR #26384: URL: https://github.com/apache/airflow/pull/26384#discussion_r971116737 ## airflow/models/baseoperator.py: ## @@ -1738,11 +1738,19 @@ def cross_downstream( task.set_downstream(to_tasks) +# pyupgrade assumes all type

[GitHub] [airflow] dstandish commented on a diff in pull request #26249: Add triggerer info to task instance in API

2022-09-14 Thread GitBox
dstandish commented on code in PR #26249: URL: https://github.com/apache/airflow/pull/26249#discussion_r971073832 ## airflow/models/taskinstance.py: ## @@ -509,14 +509,8 @@ class TaskInstance(Base, LoggingMixin): viewonly=True, ) -trigger = relationship( -

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247096211 This is actually Cool. We will test if the fix with bumping "fixed" requirements that I implemented - works. Once we merge it, new PRs should take a bit longer to build image (but they

[GitHub] [airflow] potiuk commented on pull request #26393: Bump FAB to 4.1.4

2022-09-14 Thread GitBox
potiuk commented on PR #26393: URL: https://github.com/apache/airflow/pull/26393#issuecomment-1247094363 Rebased to account for git-rev main failure :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] dimberman opened a new pull request, #26400: Create a more efficient airflow dag test command that also has better local logging

2022-09-14 Thread GitBox
dimberman opened a new pull request, #26400: URL: https://github.com/apache/airflow/pull/26400 The current `airflow dag test` command is insufficient for an iterative local workflow when developing dags. The current system relies on running a backfill job which a) is extremely slow

[GitHub] [airflow] boring-cyborg[bot] commented on issue #26399: GKEHook.create_cluster is not wait_for_operation using the input project_id parameter

2022-09-14 Thread GitBox
boring-cyborg[bot] commented on issue #26399: URL: https://github.com/apache/airflow/issues/26399#issuecomment-1247071917 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] SPTKL opened a new issue, #26399: GKEHook.create_cluster is not wait_for_operation using the input project_id parameter

2022-09-14 Thread GitBox
SPTKL opened a new issue, #26399: URL: https://github.com/apache/airflow/issues/26399 ### Apache Airflow version main (development) ### What happened In the GKEHook, the `create_cluster` method is creating a GKE cluster in the project_id specified by the input, but in

[airflow] branch main updated: Require dag_id arg for dags list-runs (#26357)

2022-09-14 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f6c579c1c0 Require dag_id arg for dags

[GitHub] [airflow] jedcunningham merged pull request #26357: Require dag_id arg for dags list-runs

2022-09-14 Thread GitBox
jedcunningham merged PR #26357: URL: https://github.com/apache/airflow/pull/26357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham commented on pull request #26357: Require dag_id arg for dags list-runs

2022-09-14 Thread GitBox
jedcunningham commented on PR #26357: URL: https://github.com/apache/airflow/pull/26357#issuecomment-1247057720 Merging as static check failures is already fixed in main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] dstandish commented on a diff in pull request #26249: Add triggerer info to task instance in API

2022-09-14 Thread GitBox
dstandish commented on code in PR #26249: URL: https://github.com/apache/airflow/pull/26249#discussion_r971073832 ## airflow/models/taskinstance.py: ## @@ -509,14 +509,8 @@ class TaskInstance(Base, LoggingMixin): viewonly=True, ) -trigger = relationship( -

[airflow] branch v2-4-test updated (b2b1416e6b -> fb89ce9dcf)

2022-09-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v2-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git from b2b1416e6b Add more to the ignore list for non-core changes (#26355) add fb89ce9dcf Work around pyupgrade

  1   2   3   >