boring-cyborg[bot] commented on pull request #10820:
URL: https://github.com/apache/airflow/pull/10820#issuecomment-689319075
Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our
anmol-dhingra opened a new pull request #10820:
URL: https://github.com/apache/airflow/pull/10820
Parameter pool was not getting passed to the Base Operator it was using
default_pool. Added it to the `options_to_remove` list so that it gets passed
as it is.
Shivarp1 edited a comment on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689279465
@kaxil I think PodDefaults in pod_generator was introduced in 1.10.12? I
have not checked the 1.10.11 version
In 1.10.10 we were using the kubernetes_request_factory,
alexbegg commented on a change in pull request #10637:
URL: https://github.com/apache/airflow/pull/10637#discussion_r485319633
##
File path: airflow/www/views.py
##
@@ -563,7 +563,9 @@ def last_dagruns(self, session=None):
return wwwutils.json_response({})
Shivarp1 commented on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689279465
@kaxil I think PodDefaults in pod_generator was introduced in 1.10.12? I
have not checked the 1.10.11 version
In 1.10.10 we were using the kubernetes_request_factory.
dimon222 edited a comment on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689229438
@kaxil I can confirm that it didn't. It might be old outstanding issue back
from 1.10.9 or even pre.
This
dimon222 edited a comment on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689229438
@kaxil I can confirm that it didn't. It might old outstanding issue back
from 1.10.9
This is an automated
dimon222 commented on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689229438
@kaxil I can confirm that it didn't. It might old outstanding bug back from
1.10.9
This is an automated message
kaxil opened a new pull request #10819:
URL: https://github.com/apache/airflow/pull/10819
`SchedulerJob.execute_helper` was renamed to
`SchedulerJob._run_scheduler_loop`
---
**^ Add meaningful description above**
Read the **[Pull Request
atsalolikhin-spokeo commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-689200729
@mik-laj Thank you for your kind comment and direction. I've fixed the code
so the CI checks are happy, and rebased to tip of master.
kaxil commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689195627
@gardnerdev Can you confirm if this worked before Airflow 1.10.12?
cc @dimberman
This is an automated
kaxil commented on issue #10722:
URL: https://github.com/apache/airflow/issues/10722#issuecomment-689195510
@Shivarp1 Can you confirm if this worked before Airflow 1.10.12?
This is an automated message from the Apache Git
kaxil opened a new pull request #10818:
URL: https://github.com/apache/airflow/pull/10818
We got clarification in https://github.com/psf/black/issues/1667 that the
new changes related to trailing commas are feature instead of a bug
---
**^ Add meaningful
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192520#comment-17192520
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
kaxil edited a comment on pull request #5499:
URL:
kaxil edited a comment on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689173053
This is an awesome feature so thank you, @KevinYang21 and other folks from
Airbnb team who worked on it.
The follow-up PRs (that I can think of right now) based on
This is an automated email from the ASF dual-hosted git repository.
dimberman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 20481c3 Add pod_override setting for
kaxil commented on issue #10817:
URL: https://github.com/apache/airflow/issues/10817#issuecomment-689179160
cc @KevinYang21
This is an automated message from the Apache Git Service.
To respond to the message, please log on
dimberman merged pull request #10756:
URL: https://github.com/apache/airflow/pull/10756
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
kaxil opened a new issue #10817:
URL: https://github.com/apache/airflow/issues/10817
An enhancement to Smart Sensor feature introduced in
https://github.com/apache/airflow/pull/5499 might be to have a separate “mode”
like "reschedule mode".
This would also simplify making Sensors
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192519#comment-17192519
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
kaxil edited a comment on pull request #5499:
URL:
kaxil commented on issue #10816:
URL: https://github.com/apache/airflow/issues/10816#issuecomment-689177720
cc @YingboWang @KevinYang21
This is an automated message from the Apache Git Service.
To respond to the message,
kaxil opened a new issue #10816:
URL: https://github.com/apache/airflow/issues/10816
**Description**
It would be good to add docs to explain different modes for Sensors:
1. Poke mode
1. Reschedule mode
1. Smart Sensor
and to explain the advantages of one over the
kaxil edited a comment on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689173053
This is an awesome feature so thank you, @KevinYang21 and other folks from
Airbnb team who worked on it.
The follow-up PRs (that I can think of right now) based on
kaxil edited a comment on issue #10815:
URL: https://github.com/apache/airflow/issues/10815#issuecomment-689174895
@YingboWang @KevinYang21 Let me if you would like to do it. If not, we
could mark this as a "good first issue" :)
kaxil commented on issue #10815:
URL: https://github.com/apache/airflow/issues/10815#issuecomment-689174895
@YingboWang Let me if you would love to do it
This is an automated message from the Apache Git Service.
To respond
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192518#comment-17192518
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
kaxil edited a comment on pull request #5499:
URL:
kaxil edited a comment on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689173053
This is an awesome feature so thank you, @KevinYang21 and other folks from
Airbnb team who worked on it.
The follow-up PRs (that I can think of right now) based on
kaxil opened a new issue #10815:
URL: https://github.com/apache/airflow/issues/10815
**Description**
Based on our discussion during Airflow 2.0 dev call: Smart Sensors Will be
included in 2.0 as an early-access feature with a clear note that this feature
might potentially change in
kaxil commented on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689173053
This is an awesome feature so thank you, @KevinYang21 and other folks from
Airbnb team who worked on it.
The follow-up PRs (that I can think of right now) based on Airflow
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192517#comment-17192517
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
kaxil commented on pull request #5499:
URL:
kenjihiraoka commented on pull request #9943:
URL: https://github.com/apache/airflow/pull/9943#issuecomment-689170646
Sorry man, I'll fix that wrong rebase :sweat:
This is an automated message from the Apache Git Service.
ephraimbuddy opened a new pull request #10814:
URL: https://github.com/apache/airflow/pull/10814
This PR adds AzureDataLakeUploadOperator. This operator will help to add
system test for ADLSToGCSOperator
---
**^ Add meaningful description above**
Read the **[Pull Request
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192510#comment-17192510
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
potiuk commented on pull request #5499:
URL:
potiuk commented on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689164512
:tada: !
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3964.
-
Resolution: Fixed
> Consolidate and de-duplicate sensor tasks
>
YingboWang commented on pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#issuecomment-689162322
> If anyone has any more suggestions or want to request changes, let's do it
in a follow-up PR.
>
> Thanks alot @YingboWang and apologies for the long wait.
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192506#comment-17192506
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
YingboWang commented on pull request #5499:
URL:
nikste commented on issue #7907:
URL: https://github.com/apache/airflow/issues/7907#issuecomment-689160101
any news on this?
This is an automated message from the Apache Git Service.
To respond to the message, please log
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192499#comment-17192499
]
ASF subversion and git services commented on AIRFLOW-3964:
--
Commit
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192497#comment-17192497
]
ASF subversion and git services commented on AIRFLOW-3964:
--
Commit
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new ac943c9 [AIRFLOW-3964][AIP-17] Consolidate
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192495#comment-17192495
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
kaxil merged pull request #5499:
URL:
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new ac943c9 [AIRFLOW-3964][AIP-17] Consolidate
kaxil merged pull request #5499:
URL: https://github.com/apache/airflow/pull/5499
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
ashb commented on pull request #10806:
URL: https://github.com/apache/airflow/pull/10806#issuecomment-689143748
> Hey @ash - I cancelled it due to "watching the watchers problem" so it's
best you rebase it :)
:+1: Done.
potiuk opened a new pull request #10813:
URL: https://github.com/apache/airflow/pull/10813
Part of #10576
---
**^ Add meaningful description above**
Read the **[Pull Request
potiuk opened a new pull request #10812:
URL: https://github.com/apache/airflow/pull/10812
Part of #10576
---
**^ Add meaningful description above**
Read the **[Pull Request
jedcunningham commented on a change in pull request #10663:
URL: https://github.com/apache/airflow/pull/10663#discussion_r485198197
##
File path: airflow/models/dagbag.py
##
@@ -269,7 +270,12 @@ def _load_modules_from_file(self, filepath, safe_mode):
return
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689133928
So, I've set retry_delay to 10s. On scheduler restart the task is stucked in
"running" state for ~4 minutes (while being "completed" on kubernetes side
before
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689124323
@dimberman You were right ! After ~10 minutes it got picked out of
"up_for_retry" state.
I guess I was a bit confused by the logs showing that the scheduler is
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689133928
So, I've set retry_delay to 10s. On scheduler restart the task is stucked in
"running" state for ~4 minutes (while being "completed" on kubernetes side
before scheduler
potiuk opened a new pull request #10811:
URL: https://github.com/apache/airflow/pull/10811
Part of #10576
---
**^ Add meaningful description above**
Read the **[Pull Request
potiuk opened a new pull request #10810:
URL: https://github.com/apache/airflow/pull/10810
Part of #10576
---
**^ Add meaningful description above**
Read the **[Pull Request
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689127606
And the default `retry_delay` seems to be 300s so everything seems to be ok.
Let's just try with a shorter retry delay.
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689124323
@dimberman You were right ! After ~10 minutes it got picked out of
"up_for_retry" state.
I guess I was a bit confused by the logs showing that the scheduler is
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689124323
@dimberman You were right ! After ~10 minutes it got picked out of
"up_for_retry" state.
I guess I was a bit confused by the logs showing that the scheduler is
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689124323
@dimberman You were right ! After ~10 minutes it got picked out of
"up_for_retry" state.
I guess I was a bit confused by the logs showing that it the scheduler is
potiuk commented on pull request #10708:
URL: https://github.com/apache/airflow/pull/10708#issuecomment-689124114
Also only quarantined test failed.
This is an automated message from the Apache Git Service.
To respond to the
potiuk commented on pull request #10734:
URL: https://github.com/apache/airflow/pull/10734#issuecomment-689123857
Just quarantined tests failed :)
This is an automated message from the Apache Git Service.
To respond to the
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689114819
> @FloChehab Ok that's a good sign (thank you btw). One more question, have
you tried leaving the task in `up_for_retry` and seeing if the scheduler
eventually picks
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689114819
> @FloChehab Ok that's a good sign (thank you btw). One more question, have
you tried leaving the task in `up_for_retry` and seeing if the scheduler
eventually picks it up?
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689113055
@FloChehab Ok that's a good sign (thank you btw). One more question, have
you tried leaving the task in `up_for_retry` and seeing if the scheduler
eventually picks it up?
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689112499
And the scheduler logs on restart:
[scheduler.log](https://github.com/apache/airflow/files/5190839/scheduler.log)
@dimberman I have to stop my investigations for
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689111872
So this time with image from v1-10-test + helm + KEDA:
* Task is stuck on running on scheduler restart (no tasks are queued on
redis)
* If run + ignore all deps =>
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 7fd65d7 Don't include kubernetes_tests/ and backport_packages/ in our
wheel (#10805)
add ff41361 Add
mik-laj merged pull request #10771:
URL: https://github.com/apache/airflow/pull/10771
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689106510
Just tested with 1.10.12 (while the image is building) and
is_delete_operator_pod=false. This time the task seemed stucked in running on
first scheduler restart. And I got
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689101838
(just need a bit more time to build the production image for 1.10-test)
This is an automated message from
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689100097
Ok let's see :)
This is an automated message from the Apache Git Service.
To respond to the message, please
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689099567
@FloChehab what happens if you are running this with the helm chart, you get
to the "up_for_retry" state, and then you manually rerun the task with "ignore
all deps"
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689097501
So, with `is_delete_operator_pod=False` and doing the same process
(including manually killing the zombie process), I do have the bug I was
describing: it took me 4
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 2811851 Move Impersonation test back to quarantine (#10809)
add 7fd65d7 Don't include kubernetes_tests/ and
potiuk commented on pull request #10806:
URL: https://github.com/apache/airflow/pull/10806#issuecomment-689097218
Hey @ash - I cancelled it due to "watching the watchers problem" so it's
best you rebase it :)
This is an
potiuk merged pull request #10805:
URL: https://github.com/apache/airflow/pull/10805
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
potiuk commented on pull request #10784:
URL: https://github.com/apache/airflow/pull/10784#issuecomment-689095091
Yeah. I noticed some unexpected behaviour with changes from yesterday and
had to fix it first :).
Pushed changes now, I also test it on my own fork (it can only be fully
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 2811851 Move Impersonation test back to
potiuk merged pull request #10809:
URL: https://github.com/apache/airflow/pull/10809
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689087705
Yeah agreed. For now if you set is_delete_operator_pod to false it fixes it.
This is an automated message
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689087054
Hum, I am not sure I would do that. I think that the life of the worker /
"object" that is starting / monitoring / etc. the pod shouldn't impact the pod
itself (we have
feluelle commented on a change in pull request #10784:
URL: https://github.com/apache/airflow/pull/10784#discussion_r485144932
##
File path: .github/workflows/ci.yml
##
@@ -35,7 +35,7 @@ env:
SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
-
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689085620
so if it recieves an error from a SIGTERM it deletes the pod because of
`is_delete_operator_pod`
This is
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689085429
Oh wait it's not the on_kill.
It's these lines
```
try:
launcher.start_pod(
pod,
kaxil commented on pull request #9847:
URL: https://github.com/apache/airflow/pull/9847#issuecomment-689083911
any updates?
This is an automated message from the Apache Git Service.
To respond to the message, please log on
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689082795
So ok, funny enough, I think because we added an on_kill to the
KubernetesPodOperator, it now kills the pod if the process dies. Not sure if
that counts as a solution or
dimberman commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689079839
Hmm... this might have to do with airflow leaving behind a zombie process,
so it's harder to get a real interruption when running locally. Will test that
now.
FloChehab edited a comment on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689078932
So I have something a bit magical going on:
* Same setup as you,
* Same process.
However I don't even need to restart the webserver or the scheduler:
turbaszek commented on a change in pull request #10802:
URL: https://github.com/apache/airflow/pull/10802#discussion_r485137457
##
File path: README.md
##
@@ -132,20 +132,13 @@ Other ways of retrieving source code are "convenience"
methods. For example, tag
> Note: Airflow
FloChehab commented on pull request #10230:
URL: https://github.com/apache/airflow/pull/10230#issuecomment-689078932
So I have something a bit magical going on:
* Same setup as you,
* Same process.
However I don't even need to restart the webserver or the scheduler:
* I
potiuk commented on pull request #10809:
URL: https://github.com/apache/airflow/pull/10809#issuecomment-689075876
Test Impersonation is failing far too often even in isolation. Moving it
back for now. :(
This is an
potiuk opened a new pull request #10809:
URL: https://github.com/apache/airflow/pull/10809
Seems that TestImpersonation is not stable even in isolation
Moving it back to quarantine for now.
---
**^ Add meaningful description above**
Read the **[Pull Request
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 3c6fdd8 Make ci/backport_packages Google Shell guide compliant
(#10733)
add c60fccc Fix integration tests
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 3c6fdd8 Make ci/backport_packages Google Shell guide compliant
(#10733)
add c60fccc Fix integration tests
potiuk merged pull request #10807:
URL: https://github.com/apache/airflow/pull/10807
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
marcjimz commented on pull request #10808:
URL: https://github.com/apache/airflow/pull/10808#issuecomment-689059424
Hi @mik-laj , thoughts on this one?
This is an automated message from the Apache Git Service.
To respond to
potiuk commented on pull request #10807:
URL: https://github.com/apache/airflow/pull/10807#issuecomment-689056179
> Quis custodiet ipsos custodes? :)
That's why we'll never run out of work. There will always be some layer
where human is needed :)
ashb commented on pull request #10807:
URL: https://github.com/apache/airflow/pull/10807#issuecomment-689055297
Quis custodiet ipsos custodes? :)
This is an automated message from the Apache Git Service.
To respond to the
marcjimz commented on pull request #10808:
URL: https://github.com/apache/airflow/pull/10808#issuecomment-689055075
I have made the equivalent changes for Airflow 2.0 and it is available here:
https://github.com/marcjimz/airflow/blob/master/airflow/providers/microsoft/azure/hooks/wasb.py
boring-cyborg[bot] commented on pull request #10808:
URL: https://github.com/apache/airflow/pull/10808#issuecomment-689054826
Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our
marcjimz opened a new pull request #10808:
URL: https://github.com/apache/airflow/pull/10808
This PR addresses the following changes:
1. Write a file from path to Azure Blob storage (WASBS)
2. Write a file from text to Azure Blob storage (WASBS)
3. Allow for kwargs to be passed
potiuk opened a new pull request #10807:
URL: https://github.com/apache/airflow/pull/10807
The change from #10769 accidentally switched Integration tests
into far-longer run unit tests (we effectively run the tests
twice and did not run integration tests.
This fixes the problem
1 - 100 of 257 matches
Mail list logo
