[jira] [Commented] (AIRFLOW-3674) Adding documentation on official docker images
[ https://issues.apache.org/jira/browse/AIRFLOW-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069244#comment-17069244 ] Jarek Potiuk commented on AIRFLOW-3674: --- I thin that one can be closed. We already have quite a documentation and more of it is coming with the production image. > Adding documentation on official docker images > -- > > Key: AIRFLOW-3674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3674 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze >Reporter: Peter van 't Hof >Assignee: Jarek Potiuk >Priority: Major > Labels: docker, dockerfile > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3674) Adding documentation on official docker images
[ https://issues.apache.org/jira/browse/AIRFLOW-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk closed AIRFLOW-3674. - Resolution: Done > Adding documentation on official docker images > -- > > Key: AIRFLOW-3674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3674 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze >Reporter: Peter van 't Hof >Assignee: Jarek Potiuk >Priority: Major > Labels: docker, dockerfile > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-265) Custom parameters for DockerOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069243#comment-17069243 ] Jarek Potiuk commented on AIRFLOW-265: -- Feel free [~Becky] > Custom parameters for DockerOperator > > > Key: AIRFLOW-265 > URL: https://issues.apache.org/jira/browse/AIRFLOW-265 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Alexandr Nikitin >Assignee: Ngwe Becky >Priority: Major > Labels: docker, gsoc, gsoc2020, mentor > > Add ability to specify custom parameters to docker cli. E.g. > "--volume-driver=""" or --net="bridge" or any other -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-265) Custom parameters for DockerOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-265: Assignee: Ngwe Becky > Custom parameters for DockerOperator > > > Key: AIRFLOW-265 > URL: https://issues.apache.org/jira/browse/AIRFLOW-265 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Alexandr Nikitin >Assignee: Ngwe Becky >Priority: Major > Labels: docker, gsoc, gsoc2020, mentor > > Add ability to specify custom parameters to docker cli. E.g. > "--volume-driver=""" or --net="bridge" or any other -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io commented on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
codecov-io edited a comment on issue #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944#issuecomment-605397082 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=h1) Report > Merging [#7944](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cb1e6c2d19b7c4233456d08d5d652ede2bae999=desc) will **increase** coverage by `0.50%`. > The diff coverage is `96.87%`. [](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7944 +/- ## == + Coverage 86.36% 86.86% +0.50% == Files 929 929 Lines 4503645074 +38 == + Hits3889539154 +259 + Misses 6141 5920 -221 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.87% <0.00%> (ø)` | | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.08% <ø> (+0.08%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <ø> (ø)` | | | [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `89.47% <80.00%> (-4.65%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `92.56% <100.00%> (+0.82%)` | :arrow_up: | | [airflow/hooks/filesystem.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9maWxlc3lzdGVtLnB5) | `90.90% <100.00%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <100.00%> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.83% <100.00%> (+0.08%)` | :arrow_up: | | [airflow/models/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2VyaWFsaXplZF9kYWcucHk=) | `94.18% <100.00%> (+1.78%)` | :arrow_up: | | ... and [31 more](https://codecov.io/gh/apache/airflow/pull/7944/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=footer). Last update [da46243...c89534d](https://codecov.io/gh/apache/airflow/pull/7944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7923: Get Airflow Variables from Environment Variables
kaxil commented on a change in pull request #7923: Get Airflow Variables from Environment Variables URL: https://github.com/apache/airflow/pull/7923#discussion_r399615781 ## File path: airflow/secrets/metastore.py ## @@ -37,3 +37,16 @@ def get_connections(self, conn_id, session=None) -> List[Connection]: conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all() session.expunge_all() return conn_list + +@provide_session +def get_variable(self, key: str, session=None): +""" +Get Airflow Variable from Metadata DB + +:param key: Variable Key +:return: Variable Value +""" +from airflow.models.variable import Variable Review comment: This looks tough as we need Variable model here and we need to get the value in Variables.py file to keep it fully backward compatible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7923: Get Airflow Variables from Environment Variables
kaxil commented on issue #7923: Get Airflow Variables from Environment Variables URL: https://github.com/apache/airflow/pull/7923#issuecomment-605390632 Separated PRs into following so that it is easier to review, merge and cherry-pick: - Hashicorp Vault: https://github.com/apache/airflow/pull/7944 - AWS Secrets Manager: https://github.com/apache/airflow/pull/7945 - GCP Secrets Manager: https://github.com/apache/airflow/pull/7946 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7913: [AIRFLOW-7113] fix gantt render error
kaxil commented on issue #7913: [AIRFLOW-7113] fix gantt render error URL: https://github.com/apache/airflow/pull/7913#issuecomment-605390525 > @KevinYang21, yes it's correct. > > @kaxil. Tasks fail at early stage, e.g. during parsing, won't contain the start date. When handling failures, scheduler will store those failed tasks in task_fail table. Which crashes the gantt chart. To reproduce the error, you need to have a row w/o start_date in task_fail table. Entries w/o start_date in task_instance table will be fine as the query filters them out. I reverted the change on it. Shouldn't we change the Scheduler code to store the start_date and if there is any error store start_date = end_date during that time before we store in task_fail table? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #7946: Get Airflow Variables from GCP Secrets Manager
kaxil opened a new pull request #7946: Get Airflow Variables from GCP Secrets Manager URL: https://github.com/apache/airflow/pull/7946 Depends on #7923 & #7944 Get Airflow Variables from AWS Systems Manager Parameter Store Please only check the last commit --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #7941: Webserver does not detach from console when using -D
zhongjiajie commented on issue #7941: Webserver does not detach from console when using -D URL: https://github.com/apache/airflow/issues/7941#issuecomment-605388696 @dimberman Hi, I found out we have so many duplicate issue, such as this one and https://github.com/apache/airflow/issues/7942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] cong-zhu commented on issue #7913: [AIRFLOW-7113] fix gantt render error
cong-zhu commented on issue #7913: [AIRFLOW-7113] fix gantt render error URL: https://github.com/apache/airflow/pull/7913#issuecomment-605388658 @KevinYang21, yes it's correct. @kaxil. Tasks fail at early stage, e.g. during parsing, won't contain the start date. When handling failures, scheduler will store those failed tasks in task_fail table. Which crashes the gantt chart. To reproduce the error, you need to have a row w/o start_date in task_fail table. Entries w/o start_date in task_instance table will be fine as the query filters them out. I reverted the change on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #7905: Pandas version pinned to < 1.0.0
zhongjiajie commented on issue #7905: Pandas version pinned to < 1.0.0 URL: https://github.com/apache/airflow/issues/7905#issuecomment-605388517 Good point, I find out Pandas < 1.0.0 release in October 31, 2019 https://pandas.pydata.org/docs/whatsnew/v0.25.3.html, and I think is new enough, although I know that 1.0.0 is the big version change. And maybe you could create some draft PR to upgrade the version, due to have experience on your daily usage @JPFrancoia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #7945: [Depends on #7923 & #7944]Get Airflow Variables from AWS Systems Manager Parameter Store
kaxil opened a new pull request #7945: [Depends on #7923 & #7944]Get Airflow Variables from AWS Systems Manager Parameter Store URL: https://github.com/apache/airflow/pull/7945 Depends on #7923 & #7944 Get Airflow Variables from AWS Systems Manager Parameter Store **_Please only check the last commit_** Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault
kaxil opened a new pull request #7944: [Depends on #7923] Get Airflow Variables from Hashicorp Vault URL: https://github.com/apache/airflow/pull/7944 Depends on #7923 Get Airflow Variables from Hashicorp Vault Please only check the last commit Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5825) SageMakerEndpointOperator is not idempotent
[ https://issues.apache.org/jira/browse/AIRFLOW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069169#comment-17069169 ] ASF subversion and git services commented on AIRFLOW-5825: -- Commit 438da7241eb537e3ef5ae711629446155bf738a3 in airflow's branch refs/heads/master from Omair Khan [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=438da72 ] [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent (#7891) > SageMakerEndpointOperator is not idempotent > --- > > Key: AIRFLOW-5825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5825 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0, 1.10.7 >Reporter: Bas Harenslak >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > The SageMakerEndpointOperator currently taken an argument "operati on" with > value "create"/"update" which determines whether to create or update a > SageMaker endpoint. However this doesn't work in the following situation: > * DAG run #1 create the endpoint (have to provide operation="create" here) > * Following DAG runs will update the endpoint created by DAG run #1 (would > have to edit DAG and set operation="update" here) > Which should be a very valid use case IMO. > The SageMakerEndpointOperator should check itself if an endpoint with name X > already exists and overwrite it (configurable desired by the user). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5825) SageMakerEndpointOperator is not idempotent
[ https://issues.apache.org/jira/browse/AIRFLOW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-5825. - Fix Version/s: 2.0.0 Resolution: Fixed > SageMakerEndpointOperator is not idempotent > --- > > Key: AIRFLOW-5825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5825 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0, 1.10.7 >Reporter: Bas Harenslak >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > The SageMakerEndpointOperator currently taken an argument "operati on" with > value "create"/"update" which determines whether to create or update a > SageMaker endpoint. However this doesn't work in the following situation: > * DAG run #1 create the endpoint (have to provide operation="create" here) > * Following DAG runs will update the endpoint created by DAG run #1 (would > have to edit DAG and set operation="update" here) > Which should be a very valid use case IMO. > The SageMakerEndpointOperator should check itself if an endpoint with name X > already exists and overwrite it (configurable desired by the user). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5825) SageMakerEndpointOperator is not idempotent
[ https://issues.apache.org/jira/browse/AIRFLOW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069168#comment-17069168 ] ASF GitHub Bot commented on AIRFLOW-5825: - kaxil commented on pull request #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent URL: https://github.com/apache/airflow/pull/7891 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > SageMakerEndpointOperator is not idempotent > --- > > Key: AIRFLOW-5825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5825 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0, 1.10.7 >Reporter: Bas Harenslak >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > > The SageMakerEndpointOperator currently taken an argument "operati on" with > value "create"/"update" which determines whether to create or update a > SageMaker endpoint. However this doesn't work in the following situation: > * DAG run #1 create the endpoint (have to provide operation="create" here) > * Following DAG runs will update the endpoint created by DAG run #1 (would > have to edit DAG and set operation="update" here) > Which should be a very valid use case IMO. > The SageMakerEndpointOperator should check itself if an endpoint with name X > already exists and overwrite it (configurable desired by the user). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent
kaxil merged pull request #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent URL: https://github.com/apache/airflow/pull/7891 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent
codecov-io edited a comment on issue #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent URL: https://github.com/apache/airflow/pull/7891#issuecomment-605375211 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=h1) Report > Merging [#7891](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/bf4d231c7402d61a3e9cd796e23fa61d24883357=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7891 +/- ## == - Coverage 87.11% 86.36% -0.75% == Files 929 929 Lines 4503645048 +12 == - Hits3923138904 -327 - Misses 5805 6144 +339 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=tree) | Coverage Δ | | |---|---|---| | [...oviders/amazon/aws/operators/sagemaker\_endpoint.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX2VuZHBvaW50LnB5) | `89.28% <100.00%> (+1.53%)` | :arrow_up: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: | | ... and [14 more](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=footer). Last update [bf4d231...7bdae44](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent
codecov-io commented on issue #7891: [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent URL: https://github.com/apache/airflow/pull/7891#issuecomment-605375211 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=h1) Report > Merging [#7891](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/bf4d231c7402d61a3e9cd796e23fa61d24883357=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7891 +/- ## == - Coverage 87.11% 86.36% -0.75% == Files 929 929 Lines 4503645048 +12 == - Hits3923138904 -327 - Misses 5805 6144 +339 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=tree) | Coverage Δ | | |---|---|---| | [...oviders/amazon/aws/operators/sagemaker\_endpoint.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX2VuZHBvaW50LnB5) | `89.28% <100.00%> (+1.53%)` | :arrow_up: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: | | ... and [14 more](https://codecov.io/gh/apache/airflow/pull/7891/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=footer). Last update [bf4d231...7bdae44](https://codecov.io/gh/apache/airflow/pull/7891?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KevinYang21 commented on issue #7792: [AIRFLOW-7102] Add try_number to airflow context
KevinYang21 commented on issue #7792: [AIRFLOW-7102] Add try_number to airflow context URL: https://github.com/apache/airflow/pull/7792#issuecomment-605367829 I feel it can go either way, maybe a bit more clean to put that into a seperate commit. @potiuk would love to hear your opinion on where should this piece of doc lives, and if possible how it should look like. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6574) Docker operator needs a private environment dict
[ https://issues.apache.org/jira/browse/AIRFLOW-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069149#comment-17069149 ] Yousef Abu-Salah commented on AIRFLOW-6574: --- Can I try doing this? > Docker operator needs a private environment dict > > > Key: AIRFLOW-6574 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6574 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Ashton Hudson >Assignee: Ashton Hudson >Priority: Major > Labels: beginner, easyfix, newbie > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The docker operator currently assigns the environment dict to the operators > self, which leads to the effect that when viewing the task in the browser, it > displays all the environment variables. > This is an issue if the docker container gets it's database credentials via > the environment variables. > A proposed solution is to create a private_environment dict that is added to > the operator's class with a leading underscore. Since the browser renderer > excludes all class attributes with a leading underscore - the information > won't be leaked. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-495) SSH Hook Problem - Failed to create remote temp file
[ https://issues.apache.org/jira/browse/AIRFLOW-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-495. --- Resolution: Auto Closed > SSH Hook Problem - Failed to create remote temp file > > > Key: AIRFLOW-495 > URL: https://issues.apache.org/jira/browse/AIRFLOW-495 > Project: Apache Airflow > Issue Type: Bug >Reporter: Jonas F >Priority: Major > Attachments: Unbenannt.PNG, sshjonas.py > > > Hi @ll, > i´ve got a Problem with Airflow SSH Hook. When i start my dag, i got an > error, that Airflow "Failed to create remote temp file". I want to open a > ssh-connection via Airflow. I set all settings in Airflow Connections and my > DAG. May someone can help me ?! > THANKS. > I´m running Airflow in a docker container. > When i run this DAG (sshjonas.py) outside of a docker-container on a local > Airflow-Installation then everythink works... > BR, > Jonas -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-495) SSH Hook Problem - Failed to create remote temp file
[ https://issues.apache.org/jira/browse/AIRFLOW-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069141#comment-17069141 ] Daniel Imberman commented on AIRFLOW-495: - This issue has been moved to https://github.com/apache/airflow/issues/7943 > SSH Hook Problem - Failed to create remote temp file > > > Key: AIRFLOW-495 > URL: https://issues.apache.org/jira/browse/AIRFLOW-495 > Project: Apache Airflow > Issue Type: Bug >Reporter: Jonas F >Priority: Major > Attachments: Unbenannt.PNG, sshjonas.py > > > Hi @ll, > i´ve got a Problem with Airflow SSH Hook. When i start my dag, i got an > error, that Airflow "Failed to create remote temp file". I want to open a > ssh-connection via Airflow. I set all settings in Airflow Connections and my > DAG. May someone can help me ?! > THANKS. > I´m running Airflow in a docker container. > When i run this DAG (sshjonas.py) outside of a docker-container on a local > Airflow-Installation then everythink works... > BR, > Jonas -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7943: SSH Hook Problem - Failed to create remote temp file
dimberman opened a new issue #7943: SSH Hook Problem - Failed to create remote temp file URL: https://github.com/apache/airflow/issues/7943 **Apache Airflow version**: None given **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Hi @ll, i´ve got a Problem with Airflow SSH Hook. When i start my dag, i got an error, that Airflow "Failed to create remote temp file". I want to open a ssh-connection via Airflow. I set all settings in Airflow Connections and my DAG. May someone can help me ?! THANKS. I´m running Airflow in a docker container. When i run this DAG (sshjonas.py) outside of a docker-container on a local Airflow-Installation then everythink works... BR, Jonas **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-486) Webserver does not detach from console when using -D
[ https://issues.apache.org/jira/browse/AIRFLOW-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069140#comment-17069140 ] Daniel Imberman commented on AIRFLOW-486: - This issue has been moved to https://github.com/apache/airflow/issues/7942 > Webserver does not detach from console when using -D > > > Key: AIRFLOW-486 > URL: https://issues.apache.org/jira/browse/AIRFLOW-486 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Bolke de Bruin >Assignee: Xuan Ji Li >Priority: Major > > Since the rework of a rolling webserver restart the "airflow webserver -D" > does not detach from the console anymore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-487) Split/Chunk operator
[ https://issues.apache.org/jira/browse/AIRFLOW-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-487. --- Resolution: Auto Closed > Split/Chunk operator > > > Key: AIRFLOW-487 > URL: https://issues.apache.org/jira/browse/AIRFLOW-487 > Project: Apache Airflow > Issue Type: New Feature >Reporter: mohsen >Priority: Major > > it would be nice to have a split (python) operator, which takes a job that > returns a list and streams each item to the specified job. like this pattern: > http://www.workflowpatterns.com/patterns/control/basic/wcp2.php > also, a chunk operator that splits a list to smaller chunks is more general > solution > i don't know if i should post this feature request here, or if there is any > try on this already. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-486) Webserver does not detach from console when using -D
[ https://issues.apache.org/jira/browse/AIRFLOW-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-486. --- Resolution: Auto Closed > Webserver does not detach from console when using -D > > > Key: AIRFLOW-486 > URL: https://issues.apache.org/jira/browse/AIRFLOW-486 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Bolke de Bruin >Assignee: Xuan Ji Li >Priority: Major > > Since the rework of a rolling webserver restart the "airflow webserver -D" > does not detach from the console anymore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-486) Webserver does not detach from console when using -D
[ https://issues.apache.org/jira/browse/AIRFLOW-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069139#comment-17069139 ] Daniel Imberman commented on AIRFLOW-486: - This issue has been moved to https://github.com/apache/airflow/issues/7941 > Webserver does not detach from console when using -D > > > Key: AIRFLOW-486 > URL: https://issues.apache.org/jira/browse/AIRFLOW-486 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Bolke de Bruin >Assignee: Xuan Ji Li >Priority: Major > > Since the rework of a rolling webserver restart the "airflow webserver -D" > does not detach from the console anymore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-484) "No data available in table" message could be clearer
[ https://issues.apache.org/jira/browse/AIRFLOW-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-484. --- Resolution: Fixed > "No data available in table" message could be clearer > - > > Key: AIRFLOW-484 > URL: https://issues.apache.org/jira/browse/AIRFLOW-484 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xuan Ji Li >Assignee: Xuan Ji Li >Priority: Trivial > > This message appears if you start airflow with no dags -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7942: Webserver does not detach from console when using -D
dimberman opened a new issue #7942: Webserver does not detach from console when using -D URL: https://github.com/apache/airflow/issues/7942 **Apache Airflow version**: None given **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Since the rework of a rolling webserver restart the "airflow webserver -D" does not detach from the console anymore **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-486 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman opened a new issue #7941: Webserver does not detach from console when using -D
dimberman opened a new issue #7941: Webserver does not detach from console when using -D URL: https://github.com/apache/airflow/issues/7941 **Apache Airflow version**: 1.7.1.2 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Since the rework of a rolling webserver restart the "airflow webserver -D" does not detach from the console anymore **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-486 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-441) DagRuns are marked as failed as soon as one task fails
[
https://issues.apache.org/jira/browse/AIRFLOW-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069137#comment-17069137
]
Daniel Imberman commented on AIRFLOW-441:
-
This issue has been moved to https://github.com/apache/airflow/issues/7940
> DagRuns are marked as failed as soon as one task fails
> --
>
> Key: AIRFLOW-441
> URL: https://issues.apache.org/jira/browse/AIRFLOW-441
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jeff Balogh
>Priority: Major
>
> https://github.com/apache/incubator-airflow/pull/1514 added a
> [{{verify_integrity}}
> function|https://github.com/apache/incubator-airflow/blob/fcf645b/airflow/models.py#L3850-L3877]
> that greedily creates {{TaskInstance}} objects for all tasks in a dag.
> This does not interact well with the assumptions in the new [{{update_state}}
> function|https://github.com/apache/incubator-airflow/blob/fcf645b/airflow/models.py#L3816-L3825].
> The guard for {{if len(tis) == len(dag.active_tasks)}} is no longer
> effective; in the old world of lazily-created tasks this code would only run
> once all the tasks in the dag had run. Now it runs all the time, and as soon
> as one task in a dag run fails the whole DagRun fails. This is bad since the
> scheduler stops processing the DagRun after that.
> In retrospect, the old code was also buggy: if your dag ends with a bunch of
> Queued tasks the DagRun could be marked as failed prematurely.
> I suspect the fix is to update the guard to look at tasks where the state is
> success or failed. Otherwise we're evaluating and failing the dag based on
> up_for_retry/queued/scheduled tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-441) DagRuns are marked as failed as soon as one task fails
[
https://issues.apache.org/jira/browse/AIRFLOW-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-441.
---
Resolution: Auto Closed
> DagRuns are marked as failed as soon as one task fails
> --
>
> Key: AIRFLOW-441
> URL: https://issues.apache.org/jira/browse/AIRFLOW-441
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jeff Balogh
>Priority: Major
>
> https://github.com/apache/incubator-airflow/pull/1514 added a
> [{{verify_integrity}}
> function|https://github.com/apache/incubator-airflow/blob/fcf645b/airflow/models.py#L3850-L3877]
> that greedily creates {{TaskInstance}} objects for all tasks in a dag.
> This does not interact well with the assumptions in the new [{{update_state}}
> function|https://github.com/apache/incubator-airflow/blob/fcf645b/airflow/models.py#L3816-L3825].
> The guard for {{if len(tis) == len(dag.active_tasks)}} is no longer
> effective; in the old world of lazily-created tasks this code would only run
> once all the tasks in the dag had run. Now it runs all the time, and as soon
> as one task in a dag run fails the whole DagRun fails. This is bad since the
> scheduler stops processing the DagRun after that.
> In retrospect, the old code was also buggy: if your dag ends with a bunch of
> Queued tasks the DagRun could be marked as failed prematurely.
> I suspect the fix is to update the guard to look at tasks where the state is
> success or failed. Otherwise we're evaluating and failing the dag based on
> up_for_retry/queued/scheduled tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7940: DagRuns are marked as failed as soon as one task fails
dimberman opened a new issue #7940: DagRuns are marked as failed as soon as one task fails URL: https://github.com/apache/airflow/issues/7940 **Apache Airflow version**: 1.7.1.2 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: https://github.com/apache/incubator-airflow/pull/1514 added a verify_integrity function that greedily creates TaskInstance objects for all tasks in a dag. This does not interact well with the assumptions in the new update_state function. The guard for if len(tis) == len(dag.active_tasks) is no longer effective; in the old world of lazily-created tasks this code would only run once all the tasks in the dag had run. Now it runs all the time, and as soon as one task in a dag run fails the whole DagRun fails. This is bad since the scheduler stops processing the DagRun after that. In retrospect, the old code was also buggy: if your dag ends with a bunch of Queued tasks the DagRun could be marked as failed prematurely. I suspect the fix is to update the guard to look at tasks where the state is success or failed. Otherwise we're evaluating and failing the dag based on up_for_retry/queued/scheduled tasks. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-441 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-433) Allow tasks to be scheduled in the current schedule interval instead of previous
[
https://issues.apache.org/jira/browse/AIRFLOW-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-433.
---
Resolution: Auto Closed
> Allow tasks to be scheduled in the current schedule interval instead of
> previous
>
>
> Key: AIRFLOW-433
> URL: https://issues.apache.org/jira/browse/AIRFLOW-433
> Project: Apache Airflow
> Issue Type: New Feature
> Components: scheduler
>Reporter: Vineet Goel
>Priority: Major
>
> I understand that airflow chose to schedule dag runs for the previous
> schedule interval because of the way data pipelines for batch processing are
> usually designed. For example, currently a daily job is scheduled for a dag
> run with execution date of Monday on a Tuesday.
> However this is a bit weird for jobs that need to provide the same day of
> running as the execution_date (or {{ ds }}). I can get around this by using
> the date_add macro function for daily jobs but this workaround fails in the
> case of jobs that are scheduled to run on weekdays and should provide the
> same day in the context.
> It might be a good idea to make this configurable for a given dag as to
> whether we want the scheduler to schedule for the same interval or the
> previous. I understand by reading the code that this might be a bigger
> feature addition than it seems as this might involve major changes to how the
> scheduler works but this should be a good feature to have as airflow is a
> great tool to use for non batch processing jobs as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7939: DagRuns are marked as failed as soon as one task fails
dimberman opened a new issue #7939: DagRuns are marked as failed as soon as one task fails URL: https://github.com/apache/airflow/issues/7939 **Apache Airflow version**: 1.7.1.2 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: https://github.com/apache/incubator-airflow/pull/1514 added a verify_integrity function that greedily creates TaskInstance objects for all tasks in a dag. This does not interact well with the assumptions in the new update_state function. The guard for if len(tis) == len(dag.active_tasks) is no longer effective; in the old world of lazily-created tasks this code would only run once all the tasks in the dag had run. Now it runs all the time, and as soon as one task in a dag run fails the whole DagRun fails. This is bad since the scheduler stops processing the DagRun after that. In retrospect, the old code was also buggy: if your dag ends with a bunch of Queued tasks the DagRun could be marked as failed prematurely. I suspect the fix is to update the guard to look at tasks where the state is success or failed. Otherwise we're evaluating and failing the dag based on up_for_retry/queued/scheduled tasks. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-441 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-405) MySQL backend connection management bug (related to PythonOperator)
[ https://issues.apache.org/jira/browse/AIRFLOW-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069135#comment-17069135 ] Daniel Imberman commented on AIRFLOW-405: - This issue has been moved to https://github.com/apache/airflow/issues/7938 > MySQL backend connection management bug (related to PythonOperator) > --- > > Key: AIRFLOW-405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-405 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.7.1.2 >Reporter: Jiasi Zeng >Priority: Major > > # Environment setup > We are running Airflow 1.7.1.2 with MySQL 5.6. The `wait_timeout` of MySQL is > set at 300 seconds, which means that idle connections will go away after 300 > seconds of inactivity. To reflect this, we set SQLAlchemy's `pool_recycle` in > `airflow.cfg` to 290 seconds, which should force Airflow/SQLAlchemy to > recycle/discard connections after 290 seconds. Thus Airflow shouldn't try to > use an already-dead connection. > # Symptom > When running a PythonOperator that takes more than 300 seconds to execute, > the task would finish executing the Python callable, but ends up with error: > https://gist.github.com/garthcn/cd7bcdec12748406506f2b0710655c8b > It seems that after the Python callable finishes executing, Airflow tries to > push its return value to XCom. However, the SQL connection has gone away > while Airflow/SQLAlchemy think it's still there. > # Hypothesis > I did some investigation and think that it might be caused by not calling > `session.commit()` or `session.close()` for the DB operations before the XCom > push. As far as I know, in SQLAlchemy, if you don't close a connection and > let it be in `checked-out` state, it won't be recycled by connection pool, > and thus SQLAlchemy will try to use it again after >300 seconds (which is the > wait_timeout for MySQL in our case). This will result in a "MySQL connection > has gone away" issue. > It seems that Airflow codebase uses @provide_context decorator to help with > session open/close, and my hunch is that some functions are not using it or > misusing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-433) Allow tasks to be scheduled in the current schedule interval instead of previous
[
https://issues.apache.org/jira/browse/AIRFLOW-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069136#comment-17069136
]
Daniel Imberman commented on AIRFLOW-433:
-
closing as 5172 has this
> Allow tasks to be scheduled in the current schedule interval instead of
> previous
>
>
> Key: AIRFLOW-433
> URL: https://issues.apache.org/jira/browse/AIRFLOW-433
> Project: Apache Airflow
> Issue Type: New Feature
> Components: scheduler
>Reporter: Vineet Goel
>Priority: Major
>
> I understand that airflow chose to schedule dag runs for the previous
> schedule interval because of the way data pipelines for batch processing are
> usually designed. For example, currently a daily job is scheduled for a dag
> run with execution date of Monday on a Tuesday.
> However this is a bit weird for jobs that need to provide the same day of
> running as the execution_date (or {{ ds }}). I can get around this by using
> the date_add macro function for daily jobs but this workaround fails in the
> case of jobs that are scheduled to run on weekdays and should provide the
> same day in the context.
> It might be a good idea to make this configurable for a given dag as to
> whether we want the scheduler to schedule for the same interval or the
> previous. I understand by reading the code that this might be a bigger
> feature addition than it seems as this might involve major changes to how the
> scheduler works but this should be a good feature to have as airflow is a
> great tool to use for non batch processing jobs as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7938: MySQL backend connection management bug (related to PythonOperator)
dimberman opened a new issue #7938: MySQL backend connection management bug (related to PythonOperator) URL: https://github.com/apache/airflow/issues/7938 **Apache Airflow version**: 1.7.1.2 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Environment setup We are running Airflow 1.7.1.2 with MySQL 5.6. The `wait_timeout` of MySQL is set at 300 seconds, which means that idle connections will go away after 300 seconds of inactivity. To reflect this, we set SQLAlchemy's `pool_recycle` in `airflow.cfg` to 290 seconds, which should force Airflow/SQLAlchemy to recycle/discard connections after 290 seconds. Thus Airflow shouldn't try to use an already-dead connection. Symptom When running a PythonOperator that takes more than 300 seconds to execute, the task would finish executing the Python callable, but ends up with error: https://gist.github.com/garthcn/cd7bcdec12748406506f2b0710655c8b It seems that after the Python callable finishes executing, Airflow tries to push its return value to XCom. However, the SQL connection has gone away while Airflow/SQLAlchemy think it's still there. Hypothesis I did some investigation and think that it might be caused by not calling `session.commit()` or `session.close()` for the DB operations before the XCom push. As far as I know, in SQLAlchemy, if you don't close a connection and let it be in `checked-out` state, it won't be recycled by connection pool, and thus SQLAlchemy will try to use it again after >300 seconds (which is the wait_timeout for MySQL in our case). This will result in a "MySQL connection has gone away" issue. It seems that Airflow codebase uses @provide_context decorator to help with session open/close, and my hunch is that some functions are not using it or misusing it. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-405) MySQL backend connection management bug (related to PythonOperator)
[ https://issues.apache.org/jira/browse/AIRFLOW-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-405. --- Resolution: Auto Closed > MySQL backend connection management bug (related to PythonOperator) > --- > > Key: AIRFLOW-405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-405 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.7.1.2 >Reporter: Jiasi Zeng >Priority: Major > > # Environment setup > We are running Airflow 1.7.1.2 with MySQL 5.6. The `wait_timeout` of MySQL is > set at 300 seconds, which means that idle connections will go away after 300 > seconds of inactivity. To reflect this, we set SQLAlchemy's `pool_recycle` in > `airflow.cfg` to 290 seconds, which should force Airflow/SQLAlchemy to > recycle/discard connections after 290 seconds. Thus Airflow shouldn't try to > use an already-dead connection. > # Symptom > When running a PythonOperator that takes more than 300 seconds to execute, > the task would finish executing the Python callable, but ends up with error: > https://gist.github.com/garthcn/cd7bcdec12748406506f2b0710655c8b > It seems that after the Python callable finishes executing, Airflow tries to > push its return value to XCom. However, the SQL connection has gone away > while Airflow/SQLAlchemy think it's still there. > # Hypothesis > I did some investigation and think that it might be caused by not calling > `session.commit()` or `session.close()` for the DB operations before the XCom > push. As far as I know, in SQLAlchemy, if you don't close a connection and > let it be in `checked-out` state, it won't be recycled by connection pool, > and thus SQLAlchemy will try to use it again after >300 seconds (which is the > wait_timeout for MySQL in our case). This will result in a "MySQL connection > has gone away" issue. > It seems that Airflow codebase uses @provide_context decorator to help with > session open/close, and my hunch is that some functions are not using it or > misusing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-405) MySQL backend connection management bug (related to PythonOperator)
[ https://issues.apache.org/jira/browse/AIRFLOW-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069133#comment-17069133 ] Daniel Imberman commented on AIRFLOW-405: - This issue has been moved to https://github.com/apache/airflow/issues/7937 > MySQL backend connection management bug (related to PythonOperator) > --- > > Key: AIRFLOW-405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-405 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.7.1.2 >Reporter: Jiasi Zeng >Priority: Major > > # Environment setup > We are running Airflow 1.7.1.2 with MySQL 5.6. The `wait_timeout` of MySQL is > set at 300 seconds, which means that idle connections will go away after 300 > seconds of inactivity. To reflect this, we set SQLAlchemy's `pool_recycle` in > `airflow.cfg` to 290 seconds, which should force Airflow/SQLAlchemy to > recycle/discard connections after 290 seconds. Thus Airflow shouldn't try to > use an already-dead connection. > # Symptom > When running a PythonOperator that takes more than 300 seconds to execute, > the task would finish executing the Python callable, but ends up with error: > https://gist.github.com/garthcn/cd7bcdec12748406506f2b0710655c8b > It seems that after the Python callable finishes executing, Airflow tries to > push its return value to XCom. However, the SQL connection has gone away > while Airflow/SQLAlchemy think it's still there. > # Hypothesis > I did some investigation and think that it might be caused by not calling > `session.commit()` or `session.close()` for the DB operations before the XCom > push. As far as I know, in SQLAlchemy, if you don't close a connection and > let it be in `checked-out` state, it won't be recycled by connection pool, > and thus SQLAlchemy will try to use it again after >300 seconds (which is the > wait_timeout for MySQL in our case). This will result in a "MySQL connection > has gone away" issue. > It seems that Airflow codebase uses @provide_context decorator to help with > session open/close, and my hunch is that some functions are not using it or > misusing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7937: MySQL backend connection management bug (related to PythonOperator)
dimberman opened a new issue #7937: MySQL backend connection management bug (related to PythonOperator) URL: https://github.com/apache/airflow/issues/7937 **Apache Airflow version**: null **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Environment setup We are running Airflow 1.7.1.2 with MySQL 5.6. The `wait_timeout` of MySQL is set at 300 seconds, which means that idle connections will go away after 300 seconds of inactivity. To reflect this, we set SQLAlchemy's `pool_recycle` in `airflow.cfg` to 290 seconds, which should force Airflow/SQLAlchemy to recycle/discard connections after 290 seconds. Thus Airflow shouldn't try to use an already-dead connection. Symptom When running a PythonOperator that takes more than 300 seconds to execute, the task would finish executing the Python callable, but ends up with error: https://gist.github.com/garthcn/cd7bcdec12748406506f2b0710655c8b It seems that after the Python callable finishes executing, Airflow tries to push its return value to XCom. However, the SQL connection has gone away while Airflow/SQLAlchemy think it's still there. Hypothesis I did some investigation and think that it might be caused by not calling `session.commit()` or `session.close()` for the DB operations before the XCom push. As far as I know, in SQLAlchemy, if you don't close a connection and let it be in `checked-out` state, it won't be recycled by connection pool, and thus SQLAlchemy will try to use it again after >300 seconds (which is the wait_timeout for MySQL in our case). This will result in a "MySQL connection has gone away" issue. It seems that Airflow codebase uses @provide_context decorator to help with session open/close, and my hunch is that some functions are not using it or misusing it. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-305) Fix /dag_stats not to double count TI belonging to the last DAG and still running
[ https://issues.apache.org/jira/browse/AIRFLOW-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069131#comment-17069131 ] Daniel Imberman commented on AIRFLOW-305: - [~sekikn] is this still an issue? > Fix /dag_stats not to double count TI belonging to the last DAG and still > running > - > > Key: AIRFLOW-305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-305 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > > Follow-up to AIRFLOW-246. That improved /dag_stats endpoint performance, but > it happened to change the result slightly; it double counts the TI belonging > to the last DAG run if it is still running. That should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend
kaxil commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend URL: https://github.com/apache/airflow/pull/7923#discussion_r399585371 ## File path: airflow/secrets/metastore.py ## @@ -37,3 +37,16 @@ def get_connections(self, conn_id, session=None) -> List[Connection]: conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all() session.expunge_all() return conn_list + +@provide_session +def get_variable(self, key: str, session=None): +""" +Get Airflow Variable from Metadata DB + +:param key: Variable Key +:return: Variable Value +""" +from airflow.models.variable import Variable Review comment: Yup, I will try to get rid of it as soon as I have the final draft ready :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-432) Failure in importing DAG
[
https://issues.apache.org/jira/browse/AIRFLOW-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-432.
---
Resolution: Auto Closed
> Failure in importing DAG
>
>
> Key: AIRFLOW-432
> URL: https://issues.apache.org/jira/browse/AIRFLOW-432
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Reporter: Shubham
>Priority: Major
>
> Hello,
> I have a simple DAG file
> from airflow import DAG
> #Dag for bulk decline of leads
> provider_bulk_decline_dag = DAG('provider_bulk_decline_dag',
> default_args=dashboard_args, schedule_interval='30 1,13 * * *')
> #Dag for wake up IVE
> wakeup_ivr_dag = DAG('wakeup_ivr_dag', default_args=dashboard_args,
> schedule_interval='00 3 * * *')
> #Dag for monetization automatic refund
> monetization_refund_dag = DAG('monetization_refund_dag',
> default_args=dashboard_args, schedule_interval='10 17 * * *')
> #Dag for monetization automatic refund
> edit_history_dag = DAG('edit_history_dag', default_args=dashboard_args,
> schedule_interval='30 18 * * *')
> My operator file looks like this:
> from airflow import DAG
> from dags import dashboard_hourly_dag
> from dags import credit_sms_dag
> from dags import hourly_dag
> from dags import daily_sms_dag
> from dags import edit_history_dag
> from airflow.operators import BashOperator, DummyOperator, PythonOperator,
> BranchPythonOperator
> editHistoryReportTask = PythonOperator(
> task_id='edit_history_report',
> provide_context=False,
> python_callable=edit_history_report_aggregator,
> op_kwargs={'env': 'PROD'},
> dag=edit_history_dag)
> But when I start the webserver, it keeps on giving me this error:
> Broken DAG: [/home/ubuntu/airflow/operators.py] cannot import name
> edit_history_dag
> While the command aiflow list_dags displays the above dag.
> The error in log file is:
> ERROR - Failed to import: /home/ubuntu/airflow/operators.py
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 247,
> in process_file
> m = imp.load_source(mod_name, filepath)
> File "/home/ubuntu/airflow/operators.py", line 6, in
> from dags import edit_history_dag
> ImportError: cannot import name edit_history_dag
> Please help with the issue coz it reoccurs whenever I add a new dag.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-417) UI should not print traceback for missing dag/task in URL
[
https://issues.apache.org/jira/browse/AIRFLOW-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069130#comment-17069130
]
Daniel Imberman commented on AIRFLOW-417:
-
This issue has been moved to https://github.com/apache/airflow/issues/7936
> UI should not print traceback for missing dag/task in URL
> -
>
> Key: AIRFLOW-417
> URL: https://issues.apache.org/jira/browse/AIRFLOW-417
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Reporter: Dan Davydov
>Assignee: Vijay Bhat
>Priority: Major
> Labels: UI, easy-fix
>
> Right now if a user runs tries to do certain things in the UI with dags/tasks
> that don't exist they get confusing tracebacks rather than an error rendered
> in html like "the dag/task doesn't exist". One such traceback can be seen by
> going to the tree view for any DAG in the UI and then changing the url in the
> address bar for the dag_id to be a non-existent dag. The following traceback
> can be seen:
> {quote}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1817, in wsgi_app
> response = self.full_dispatch_request()
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1477, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1381, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1475, in full_dispatch_request
> rv = self.dispatch_request()
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1461, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py",
> line 68, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py",
> line 367, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Login-0.2.11-py2.7.egg/flask_login.py",
> line 758, in decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 213, in view_func
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1208, in tree
> base_date = dag.latest_execution_date or datetime.now()
> AttributeError: 'NoneType' object has no attribute 'latest_execution_date'
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7936: UI should not print traceback for missing dag/task in URL
dimberman opened a new issue #7936: UI should not print traceback for missing dag/task in URL URL: https://github.com/apache/airflow/issues/7936 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Right now if a user runs tries to do certain things in the UI with dags/tasks that don't exist they get confusing tracebacks rather than an error rendered in html like "the dag/task doesn't exist". One such traceback can be seen by going to the tree view for any DAG in the UI and then changing the url in the address bar for the dag_id to be a non-existent dag. The following traceback can be seen: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py", line 68, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py", line 367, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/Flask_Login-0.2.11-py2.7.egg/flask_login.py", line 758, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 213, in view_func return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 118, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1208, in tree base_date = dag.latest_execution_date or datetime.now() AttributeError: 'NoneType' object has no attribute 'latest_execution_date' **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-417 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-417) UI should not print traceback for missing dag/task in URL
[
https://issues.apache.org/jira/browse/AIRFLOW-417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-417.
---
Resolution: Auto Closed
> UI should not print traceback for missing dag/task in URL
> -
>
> Key: AIRFLOW-417
> URL: https://issues.apache.org/jira/browse/AIRFLOW-417
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Reporter: Dan Davydov
>Assignee: Vijay Bhat
>Priority: Major
> Labels: UI, easy-fix
>
> Right now if a user runs tries to do certain things in the UI with dags/tasks
> that don't exist they get confusing tracebacks rather than an error rendered
> in html like "the dag/task doesn't exist". One such traceback can be seen by
> going to the tree view for any DAG in the UI and then changing the url in the
> address bar for the dag_id to be a non-existent dag. The following traceback
> can be seen:
> {quote}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1817, in wsgi_app
> response = self.full_dispatch_request()
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1477, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1381, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1475, in full_dispatch_request
> rv = self.dispatch_request()
> File
> "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1461, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py",
> line 68, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Admin-1.4.0-py2.7.egg/flask_admin/base.py",
> line 367, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/Flask_Login-0.2.11-py2.7.egg/flask_login.py",
> line 758, in decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 213, in view_func
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1208, in tree
> base_date = dag.latest_execution_date or datetime.now()
> AttributeError: 'NoneType' object has no attribute 'latest_execution_date'
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-401) scheduler gets stuck without a trace
[ https://issues.apache.org/jira/browse/AIRFLOW-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-401. --- Resolution: Auto Closed > scheduler gets stuck without a trace > > > Key: AIRFLOW-401 > URL: https://issues.apache.org/jira/browse/AIRFLOW-401 > Project: Apache Airflow > Issue Type: Bug > Components: executors, scheduler >Affects Versions: 1.7.1.3 >Reporter: Nadeem Ahmed Nazeer >Assignee: Ash Berlin-Taylor >Priority: Minor > Labels: celery, kombu > Attachments: Dag_code.txt, schduler_cpu100%.png, scheduler_stuck.png, > scheduler_stuck_7hours.png > > > The scheduler gets stuck without a trace or error. When this happens, the CPU > usage of scheduler service is at 100%. No jobs get submitted and everything > comes to a halt. Looks it goes into some kind of infinite loop. > The only way I could make it run again is by manually restarting the > scheduler service. But again, after running some tasks it gets stuck. I've > tried with both Celery and Local executors but same issue occurs. I am using > the -n 3 parameter while starting scheduler. > Scheduler configs, > job_heartbeat_sec = 5 > scheduler_heartbeat_sec = 5 > executor = LocalExecutor > parallelism = 32 > Please help. I would be happy to provide any other information needed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7935: scheduler gets stuck without a trace
dimberman opened a new issue #7935: scheduler gets stuck without a trace URL: https://github.com/apache/airflow/issues/7935 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: The scheduler gets stuck without a trace or error. When this happens, the CPU usage of scheduler service is at 100%. No jobs get submitted and everything comes to a halt. Looks it goes into some kind of infinite loop. The only way I could make it run again is by manually restarting the scheduler service. But again, after running some tasks it gets stuck. I've tried with both Celery and Local executors but same issue occurs. I am using the -n 3 parameter while starting scheduler. Scheduler configs, job_heartbeat_sec = 5 scheduler_heartbeat_sec = 5 executor = LocalExecutor parallelism = 32 Please help. I would be happy to provide any other information needed **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-401 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-401) scheduler gets stuck without a trace
[ https://issues.apache.org/jira/browse/AIRFLOW-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069129#comment-17069129 ] Daniel Imberman commented on AIRFLOW-401: - This issue has been moved to https://github.com/apache/airflow/issues/7935 > scheduler gets stuck without a trace > > > Key: AIRFLOW-401 > URL: https://issues.apache.org/jira/browse/AIRFLOW-401 > Project: Apache Airflow > Issue Type: Bug > Components: executors, scheduler >Affects Versions: 1.7.1.3 >Reporter: Nadeem Ahmed Nazeer >Assignee: Ash Berlin-Taylor >Priority: Minor > Labels: celery, kombu > Attachments: Dag_code.txt, schduler_cpu100%.png, scheduler_stuck.png, > scheduler_stuck_7hours.png > > > The scheduler gets stuck without a trace or error. When this happens, the CPU > usage of scheduler service is at 100%. No jobs get submitted and everything > comes to a halt. Looks it goes into some kind of infinite loop. > The only way I could make it run again is by manually restarting the > scheduler service. But again, after running some tasks it gets stuck. I've > tried with both Celery and Local executors but same issue occurs. I am using > the -n 3 parameter while starting scheduler. > Scheduler configs, > job_heartbeat_sec = 5 > scheduler_heartbeat_sec = 5 > executor = LocalExecutor > parallelism = 32 > Please help. I would be happy to provide any other information needed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-392) DAG runs on strange schedule in the past when deployed
[
https://issues.apache.org/jira/browse/AIRFLOW-392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-392.
---
Resolution: Auto Closed
> DAG runs on strange schedule in the past when deployed
> --
>
> Key: AIRFLOW-392
> URL: https://issues.apache.org/jira/browse/AIRFLOW-392
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.7.1.3
> Environment: AWS ElasticBeanstalk as a Docker application running in
> an Ubuntu-based container
>Reporter: David Klosowski
>Assignee: Norman Mu
>Priority: Major
>
> Just deployed a new DAG ('weekly-no-track') that depends on 7 DAG task runs
> of another DAG ('daily-no-track'). When the DAG is deployed the scheduler
> schedules and runs multiple runs in the past (yesterday it ran for 6/12/2016
> and 6/05/2016), despite the start date set to the deployment date.
> It would be a bit difficult to include all the code being used in the DAG
> since we have multiple libraries we've built in Python that are being
> referenced here that we want to eventually open source. I've included some
> of the code here. Let me know if this is all clear and what I can do to help
> or if any insight can be provided as to what it is occurring and how we might
> fix this.
> {code}
> from __future__ import division, print_function
> from airflow.models import DAG
> from airflow.operators import DummyOperator, ExternalTaskSensor,
> TimeDeltaSensor
> from tn_etl_tools.aws.emr import EmrConfig, HiveConfig, read_cluster_templates
> from tn_etl_tools.aws.emr import EmrService, EmrServiceWrapper,
> HiveStepBuilder
> from tn_etl_tools.datesupport import ts_add
> from tn_etl_tools.hive import HivePartitions
> from tn_etl_tools.yaml import YamlLoader
> from datetime import datetime, timedelta
> from dateutil.relativedelta import relativedelta, SU, MO , TU, WE, TH, FR, SA
> from common_args import merge_dicts, CommonHiveParams
> from operator_builders import add_tasks, emr_hive_operator
> import os
> # === configs
> config_dir = os.getenv('DAG_CONFIG_DIR', '/usr/local/airflow/config')
> alert_email = os.getenv('AIRFLOW_TO_EMAIL')
> app_properties = YamlLoader.load_yaml(config_dir + '/app.yml')
> emr_cluster_properties = YamlLoader.load_yaml(config_dir +
> '/emr_clusters.yml')
> emr_config = EmrConfig.load(STAGE=app_properties['STAGE'],
> **app_properties['EMR'])
> hive_config = HiveConfig.load(STAGE=app_properties['STAGE'],
> **app_properties['HIVE'])
> emr_cluster_templates = read_cluster_templates(emr_cluster_properties)
> # === /configs
> # TODO: force execution_date = sunday?
> run_for_date = datetime(2016, 8, 8)
> emr_service = EmrService()
> emr_service_wrapper = EmrServiceWrapper(emr_service=emr_service,
> emr_config=emr_config,
> cluster_templates=emr_cluster_templates)
> hive_step_builder = HiveStepBuilder(hive_config=hive_config)
> hive_params = CommonHiveParams(app_properties_hive=app_properties['HIVE'])
> args = {'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': run_for_date,
> 'email': [alert_email],
> 'email_on_failure': True,
> 'email_on_retry': False,
> 'retries': 1,
> 'trigger_rule' : 'all_success',
> 'emr_service_wrapper': emr_service_wrapper,
> 'hive_step_builder': hive_step_builder}
> user_defined_macros = {'hive_partitions': HivePartitions,
>'ts_add': ts_add}
> params = {'stage': app_properties['STAGE']}
> dag = DAG(dag_id='weekly_no_track', default_args=args,
> user_defined_macros=user_defined_macros, params=params,
> schedule_interval=timedelta(days=7),
> max_active_runs=1)
> # === task definitions
> task_definitions = {
> 'wait-for-dailies': {
> 'operator_type': 'dummy_operator', # hub for custom defined
> dependencies
> 'operator_args': {},
> 'depends_on': []
> },
> 'weekly-no-track': {
> 'operator_type': 'emr_hive_operator',
> 'operator_args': {
> 'hive_step': {
> 'script': 'weekly-no-track-airflow', # temporary modified
> script with separate output path
> 'cluster_name': 'geoprofile',
> 'script_vars': merge_dicts(hive_params.default_params(),
> hive_params.rundate_params(), {
> 'PARTITIONS': '{{hive_partitions.by_day(ts_add(ts,
> days=-6), ts_add(ts, days=1))}}',
> }),
> }
> },
> 'depends_on': ['wait-for-dailies']
> }
> }
> # === /task definitions
> operator_builders = {'emr_hive_operator': emr_hive_operator,
> 'time_delta_sensor': TimeDeltaSensor,
> 'dummy_operator': DummyOperator}
[GitHub] [airflow] dimberman opened a new issue #7934: Support more general and customizable trigger rule options
dimberman opened a new issue #7934: Support more general and customizable trigger rule options URL: https://github.com/apache/airflow/issues/7934 **Description** Currently, the trigger rule options in Airflow are very limited. For example, it is difficult to let a task trigger when the upstream either failed or succeeded (but not upstream_failed). We should support more general and more customizable trigger rule options. **Use case / motivation** **Related Issues** Moved here from https://issues.apache.org/jira/browse/AIRFLOW-389 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-389) Support more general and customizable trigger rule options
[ https://issues.apache.org/jira/browse/AIRFLOW-389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069127#comment-17069127 ] Daniel Imberman commented on AIRFLOW-389: - This issue has been moved to https://github.com/apache/airflow/issues/7934 > Support more general and customizable trigger rule options > -- > > Key: AIRFLOW-389 > URL: https://issues.apache.org/jira/browse/AIRFLOW-389 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Tongzhou Wang >Assignee: Tongzhou Wang >Priority: Minor > > Currently, the trigger rule options in Airflow are very limited. For example, > it is difficult to let a task trigger when the upstream either failed or > succeeded (but not upstream_failed). > We should support more general and more customizable trigger rule options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-382) YarnExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-382. --- Resolution: Auto Closed > YarnExecutor > > > Key: AIRFLOW-382 > URL: https://issues.apache.org/jira/browse/AIRFLOW-382 > Project: Apache Airflow > Issue Type: New Feature > Components: executors >Reporter: Steve Morin >Priority: Critical > > I am looking for YarnExecutor support. I wanted to see what support there is > for this and if it's on the roadmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-389) Support more general and customizable trigger rule options
[ https://issues.apache.org/jira/browse/AIRFLOW-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-389. --- Resolution: Auto Closed > Support more general and customizable trigger rule options > -- > > Key: AIRFLOW-389 > URL: https://issues.apache.org/jira/browse/AIRFLOW-389 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Tongzhou Wang >Assignee: Tongzhou Wang >Priority: Minor > > Currently, the trigger rule options in Airflow are very limited. For example, > it is difficult to let a task trigger when the upstream either failed or > succeeded (but not upstream_failed). > We should support more general and more customizable trigger rule options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-377) Airflow does not warn if we create 2 dags with the same dag_id
[ https://issues.apache.org/jira/browse/AIRFLOW-377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-377. --- Resolution: Auto Closed > Airflow does not warn if we create 2 dags with the same dag_id > -- > > Key: AIRFLOW-377 > URL: https://issues.apache.org/jira/browse/AIRFLOW-377 > Project: Apache Airflow > Issue Type: Bug >Reporter: Xuan Ji Li >Assignee: Xuan Ji Li >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-377) Airflow does not warn if we create 2 dags with the same dag_id
[ https://issues.apache.org/jira/browse/AIRFLOW-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069126#comment-17069126 ] Daniel Imberman commented on AIRFLOW-377: - This issue has been moved to https://github.com/apache/airflow/issues/7933 > Airflow does not warn if we create 2 dags with the same dag_id > -- > > Key: AIRFLOW-377 > URL: https://issues.apache.org/jira/browse/AIRFLOW-377 > Project: Apache Airflow > Issue Type: Bug >Reporter: Xuan Ji Li >Assignee: Xuan Ji Li >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ad-m commented on issue #6625: [AIRFLOW-6027] Disable API access by default
ad-m commented on issue #6625: [AIRFLOW-6027] Disable API access by default URL: https://github.com/apache/airflow/pull/6625#issuecomment-605357600 Not, but GitHub support advisory ( https://help.github.com/en/github/managing-security-vulnerabilities/publishing-a-security-advisory ) for publishing & obtain CVE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-376) TypeError("Boolean value of this clause is not defined")
[
https://issues.apache.org/jira/browse/AIRFLOW-376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-376.
---
Resolution: Auto Closed
> TypeError("Boolean value of this clause is not defined")
>
>
> Key: AIRFLOW-376
> URL: https://issues.apache.org/jira/browse/AIRFLOW-376
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xuan Ji Li
>Assignee: Xuan Ji Li
>Priority: Minor
>
> With this dag,
> ```
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> from datetime import datetime, timedelta
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 1, 1, 1, 0),
> 'email': ['xua...@gmail.com'],
> 'email_on_failure': True,
> 'email_on_retry': False,
> 'retries': 3,
> 'retry_delay': timedelta(minutes=1),
> }
> dag = DAG('bash_bash_bash', default_args=default_args)
> # t1, t2 and t3 are examples of tasks created by instatiating operators
> t1 = BashOperator(
> task_id='print_date',
> bash_command='date',
> dag=dag)
> t2 = BashOperator(
> task_id='sleep',
> bash_command='sleep 5',
> retries=3,
> dag=dag)
> templated_command = """
> {% for i in range(5) %}
> echo "{{ ds }}"
> echo "{{ macros.ds_add(ds, 7)}}"
> echo "{{ params.my_param }}"
> {% endfor %}
> """
> t3 = BashOperator(
> task_id='templated',
> bash_command=templated_command,
> params={'my_param': 'Parameter I passed in'},
> dag=dag)
> t2.set_upstream(t1)
> t3.set_upstream(t1)
> ```
> I get an error while running the scheduler
> ```
> [2016-07-27 21:40:57,468] {jobs.py:669} ERROR - Boolean value of this clause
> is not defined
> Traceback (most recent call last):
> File "/Users/xuanji_li/tools/zodiac-airflow/airflow/jobs.py", line 667, in
> _do_dags
> self.manage_slas(dag)
> File "/Users/xuanji_li/tools/zodiac-airflow/airflow/utils/db.py", line 53,
> in wrapper
> result = func(*args, **kwargs)
> File "/Users/xuanji_li/tools/zodiac-airflow/airflow/jobs.py", line 299, in
> manage_slas
> .filter(SlaMiss.email_sent.is_(False) or
> SlaMiss.notification_sent.is_(False))
> File "/Library/Python/2.7/site-packages/sqlalchemy/sql/elements.py", line
> 2760, in __bool__
> raise TypeError("Boolean value of this clause is not defined")
> TypeError: Boolean value of this clause is not defined
> ```
> Mainly opening this to remind myself to take a look at it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7933: Airflow does not warn if we create 2 dags with the same dag_id
dimberman opened a new issue #7933: Airflow does not warn if we create 2 dags with the same dag_id URL: https://github.com/apache/airflow/issues/7933 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Click to add description **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-377 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend
mik-laj commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend URL: https://github.com/apache/airflow/pull/7923#discussion_r399583454 ## File path: airflow/secrets/metastore.py ## @@ -37,3 +37,16 @@ def get_connections(self, conn_id, session=None) -> List[Connection]: conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all() session.expunge_all() return conn_list + +@provide_session +def get_variable(self, key: str, session=None): +""" +Get Airflow Variable from Metadata DB + +:param key: Variable Key +:return: Variable Value +""" +from airflow.models.variable import Variable Review comment: Cyclical import? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend
mik-laj commented on a change in pull request #7923: [WIP] Get Airflow Connection from Environment Variables & SecretsBackend URL: https://github.com/apache/airflow/pull/7923#discussion_r399583454 ## File path: airflow/secrets/metastore.py ## @@ -37,3 +37,16 @@ def get_connections(self, conn_id, session=None) -> List[Connection]: conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all() session.expunge_all() return conn_list + +@provide_session +def get_variable(self, key: str, session=None): +""" +Get Airflow Variable from Metadata DB + +:param key: Variable Key +:return: Variable Value +""" +from airflow.models.variable import Variable Review comment: Cyclical import? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-372) DAGs can run before start_date time
[ https://issues.apache.org/jira/browse/AIRFLOW-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069121#comment-17069121 ] Daniel Imberman commented on AIRFLOW-372: - This issue has been moved to https://github.com/apache/airflow/issues/7932 > DAGs can run before start_date time > --- > > Key: AIRFLOW-372 > URL: https://issues.apache.org/jira/browse/AIRFLOW-372 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.7.1.2 >Reporter: Isaac Steele >Priority: Major > > If you turn off a DAG in the UI, there seemingly is no way to prevent > "missed" runs to schedule after the DAG is turned back on. I thought the > workaround for this, since it is not a parameterized option to prevent, would > be to update the start_date in the DAG code before turning the DAG back on. > This does not work, and therefore the scheduler is running dag_runs *before* > the listed start_date. > To reproduce: > # Create a DAG with a schedule_interval > # Let the DAG run at least once > # Turn off the DAG in the UI > # Allow the schedule_interval to pass at least twice > # Update the start_date in the DAG to be be after the two interval time > # (I then removed the compiled python file and restarted airflow/scheduler > just to make sure) > # Turn DAG back on in UI > Result: All dag_runs that were "missed" while the DAG was turned off run, > despite the start_date being later. > Ideally the start_date would always be honored. And also there would be a > parameter to just not run any "missed" dag_runs. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7932: DAGs can run before start_date time
dimberman opened a new issue #7932: DAGs can run before start_date time URL: https://github.com/apache/airflow/issues/7932 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: If you turn off a DAG in the UI, there seemingly is no way to prevent "missed" runs to schedule after the DAG is turned back on. I thought the workaround for this, since it is not a parameterized option to prevent, would be to update the start_date in the DAG code before turning the DAG back on. This does not work, and therefore the scheduler is running dag_runs before the listed start_date. To reproduce: Create a DAG with a schedule_interval Let the DAG run at least once Turn off the DAG in the UI Allow the schedule_interval to pass at least twice Update the start_date in the DAG to be be after the two interval time (I then removed the compiled python file and restarted airflow/scheduler just to make sure) Turn DAG back on in UI Result: All dag_runs that were "missed" while the DAG was turned off run, despite the start_date being later. Ideally the start_date would always be honored. And also there would be a parameter to just not run any "missed" dag_runs. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-372) DAGs can run before start_date time
[ https://issues.apache.org/jira/browse/AIRFLOW-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069120#comment-17069120 ] Daniel Imberman commented on AIRFLOW-372: - This issue has been moved to https://github.com/apache/airflow/issues/7931 > DAGs can run before start_date time > --- > > Key: AIRFLOW-372 > URL: https://issues.apache.org/jira/browse/AIRFLOW-372 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.7.1.2 >Reporter: Isaac Steele >Priority: Major > > If you turn off a DAG in the UI, there seemingly is no way to prevent > "missed" runs to schedule after the DAG is turned back on. I thought the > workaround for this, since it is not a parameterized option to prevent, would > be to update the start_date in the DAG code before turning the DAG back on. > This does not work, and therefore the scheduler is running dag_runs *before* > the listed start_date. > To reproduce: > # Create a DAG with a schedule_interval > # Let the DAG run at least once > # Turn off the DAG in the UI > # Allow the schedule_interval to pass at least twice > # Update the start_date in the DAG to be be after the two interval time > # (I then removed the compiled python file and restarted airflow/scheduler > just to make sure) > # Turn DAG back on in UI > Result: All dag_runs that were "missed" while the DAG was turned off run, > despite the start_date being later. > Ideally the start_date would always be honored. And also there would be a > parameter to just not run any "missed" dag_runs. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-372) DAGs can run before start_date time
[ https://issues.apache.org/jira/browse/AIRFLOW-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-372. --- Resolution: Auto Closed > DAGs can run before start_date time > --- > > Key: AIRFLOW-372 > URL: https://issues.apache.org/jira/browse/AIRFLOW-372 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.7.1.2 >Reporter: Isaac Steele >Priority: Major > > If you turn off a DAG in the UI, there seemingly is no way to prevent > "missed" runs to schedule after the DAG is turned back on. I thought the > workaround for this, since it is not a parameterized option to prevent, would > be to update the start_date in the DAG code before turning the DAG back on. > This does not work, and therefore the scheduler is running dag_runs *before* > the listed start_date. > To reproduce: > # Create a DAG with a schedule_interval > # Let the DAG run at least once > # Turn off the DAG in the UI > # Allow the schedule_interval to pass at least twice > # Update the start_date in the DAG to be be after the two interval time > # (I then removed the compiled python file and restarted airflow/scheduler > just to make sure) > # Turn DAG back on in UI > Result: All dag_runs that were "missed" while the DAG was turned off run, > despite the start_date being later. > Ideally the start_date would always be honored. And also there would be a > parameter to just not run any "missed" dag_runs. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-357) How should I run task with owner not shell owner in AIRFLOW
[
https://issues.apache.org/jira/browse/AIRFLOW-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-357.
---
Resolution: Auto Closed
> How should I run task with owner not shell owner in AIRFLOW
>
>
> Key: AIRFLOW-357
> URL: https://issues.apache.org/jira/browse/AIRFLOW-357
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: wei.he
>Priority: Major
>
> I dont understand the "owner" in airflow. the comment of ower is "the owner
> of the task, using the unix username is recommended". I wrote some the
> following code.
> Default_args = {
> 'owner': 'max',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 7, 14),
> 'email': ['m...@test.com'],
> 'email_on_failure': False,
> 'email_on_retry': False,
> 'retries': 1,
> 'retry_delay': timedelta(minutes=5),
> dag = DAG('dmp-annalect', default_args=default_args,
> schedule_interval='30 0 * * *')
> task1_pigjob_basedata = """
> {local_dir}/src/basedata/basedata.sh > {local_dir}/log/basedata/run_log &
> """.format(local_dir=WORKSPACE)
> task1_pigjob_basedata = BashOperator(
> task_id='task1_pigjob_basedata_impclk',owner='max',
> bash_command=pigjob_basedata_impclk,
> dag=dag)
> I used the command "airflow test dagid taskid 2016-07-20" ,
> But I got a error,
> ... {bash_operator.py:77} INFO - put: Permission denied: user=airflow,
> I thought that my job ran with "max" user, but apperently , ran test using
> 'airflow' user .
> I hope if I run my task using 'max' user, how should I do.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7931: DAGs can run before start_date time
dimberman opened a new issue #7931: DAGs can run before start_date time URL: https://github.com/apache/airflow/issues/7931 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: If you turn off a DAG in the UI, there seemingly is no way to prevent "missed" runs to schedule after the DAG is turned back on. I thought the workaround for this, since it is not a parameterized option to prevent, would be to update the start_date in the DAG code before turning the DAG back on. This does not work, and therefore the scheduler is running dag_runs before the listed start_date. To reproduce: Create a DAG with a schedule_interval Let the DAG run at least once Turn off the DAG in the UI Allow the schedule_interval to pass at least twice Update the start_date in the DAG to be be after the two interval time (I then removed the compiled python file and restarted airflow/scheduler just to make sure) Turn DAG back on in UI Result: All dag_runs that were "missed" while the DAG was turned off run, despite the start_date being later. Ideally the start_date would always be honored. And also there would be a parameter to just not run any "missed" dag_runs. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman opened a new issue #7930: How should I run task with owner not shell owner in AIRFLOW
dimberman opened a new issue #7930: How should I run task with owner not shell
owner in AIRFLOW
URL: https://github.com/apache/airflow/issues/7930
**Apache Airflow version**:
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
I dont understand the "owner" in airflow. the comment of ower is "the owner
of the task, using the unix username is recommended". I wrote some the
following code.
Default_args =
{
'owner': 'max',
'depends_on_past': False,
'start_date': datetime(2016, 7, 14),
'email': ['m...@test.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
dag = DAG('dmp-annalect', default_args=default_args,
schedule_interval='30 0 * * *')
task1_pigjob_basedata = """
{local_dir}
/src/basedata/basedata.sh >
{local_dir}
/log/basedata/run_log &
""".format(local_dir=WORKSPACE)
task1_pigjob_basedata = BashOperator(
task_id='task1_pigjob_basedata_impclk',owner='max',
bash_command=pigjob_basedata_impclk,
dag=dag)
I used the command "airflow test dagid taskid 2016-07-20" ,
But I got a error,
...
{bash_operator.py:77}
INFO - put: Permission denied: user=airflow,
I thought that my job ran with "max" user, but apperently , ran test using
'airflow' user .
I hope if I run my task using 'max' user, how should I do.
Thanks
**What you expected to happen**:
**How to reproduce it**:
**Anything else we need to know**:
Moved here from https://issues.apache.org/jira/browse/AIRFLOW-357
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-357) How should I run task with owner not shell owner in AIRFLOW
[
https://issues.apache.org/jira/browse/AIRFLOW-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069119#comment-17069119
]
Daniel Imberman commented on AIRFLOW-357:
-
This issue has been moved to https://github.com/apache/airflow/issues/7930
> How should I run task with owner not shell owner in AIRFLOW
>
>
> Key: AIRFLOW-357
> URL: https://issues.apache.org/jira/browse/AIRFLOW-357
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: wei.he
>Priority: Major
>
> I dont understand the "owner" in airflow. the comment of ower is "the owner
> of the task, using the unix username is recommended". I wrote some the
> following code.
> Default_args = {
> 'owner': 'max',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 7, 14),
> 'email': ['m...@test.com'],
> 'email_on_failure': False,
> 'email_on_retry': False,
> 'retries': 1,
> 'retry_delay': timedelta(minutes=5),
> dag = DAG('dmp-annalect', default_args=default_args,
> schedule_interval='30 0 * * *')
> task1_pigjob_basedata = """
> {local_dir}/src/basedata/basedata.sh > {local_dir}/log/basedata/run_log &
> """.format(local_dir=WORKSPACE)
> task1_pigjob_basedata = BashOperator(
> task_id='task1_pigjob_basedata_impclk',owner='max',
> bash_command=pigjob_basedata_impclk,
> dag=dag)
> I used the command "airflow test dagid taskid 2016-07-20" ,
> But I got a error,
> ... {bash_operator.py:77} INFO - put: Permission denied: user=airflow,
> I thought that my job ran with "max" user, but apperently , ran test using
> 'airflow' user .
> I hope if I run my task using 'max' user, how should I do.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-351) Failed to clear downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-351.
---
Resolution: Auto Closed
> Failed to clear downstream tasks
>
>
> Key: AIRFLOW-351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-351
> Project: Apache Airflow
> Issue Type: Bug
> Components: models, operators, webserver
>Affects Versions: 1.7.1.3
>Reporter: Adinata
>Priority: Major
> Labels: subdag
> Attachments: dag_error.py, error.log, error_on_clear_dag.txt,
> ubuntu-14-packages.log, ubuntu-16-oops.log, ubuntu-16-packages.log
>
>
> {code}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: 9889a7c79e9b
> ---
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line
> 367, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in
> decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 167, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1017, in clear
> include_upstream=upstream)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870,
> in sub_dag
> dag = copy.deepcopy(self)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
> state = deepcopy(state, memo)
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
>
[jira] [Commented] (AIRFLOW-351) Failed to clear downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069117#comment-17069117
]
Daniel Imberman commented on AIRFLOW-351:
-
This issue has been moved to https://github.com/apache/airflow/issues/7929
> Failed to clear downstream tasks
>
>
> Key: AIRFLOW-351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-351
> Project: Apache Airflow
> Issue Type: Bug
> Components: models, operators, webserver
>Affects Versions: 1.7.1.3
>Reporter: Adinata
>Priority: Major
> Labels: subdag
> Attachments: dag_error.py, error.log, error_on_clear_dag.txt,
> ubuntu-14-packages.log, ubuntu-16-oops.log, ubuntu-16-packages.log
>
>
> {code}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: 9889a7c79e9b
> ---
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line
> 367, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in
> decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 167, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1017, in clear
> include_upstream=upstream)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870,
> in sub_dag
> dag = copy.deepcopy(self)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
> state = deepcopy(state, memo)
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y =
[GitHub] [airflow] dimberman opened a new issue #7929: Failed to clear downstream tasks
dimberman opened a new issue #7929: Failed to clear downstream tasks URL: https://github.com/apache/airflow/issues/7929 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: / ( () ) \___ /( ( ( ) _)) ) )\ (( ( )() ) ( ) ) ((/ ( _( ) ( _) ) ( () ) ) ( ( ( (_) ((( ) .((_ ) . )_ ( ( )( ( )) ) . ) ( ) ( ( ( ( ) ( _ ( _) ). ) . ) ) ( ) ( ( ( ) ( ) ( )) ) _)( ) ) ) ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( ) ( ( ( ( (_ ( ) ( _) ) ( ) ) ) ( ( ( ( ( ) (_ ) ) ) _) ) _( ( ) (( ( )(( _) _) _(_ ( (_ ) (_((__(_(__(( ( ( | ) ) ) )_))__))_)___) ((__)\\||lll|l||/// \_)) ( /(/ ( ) ) )\ ) (( ( ( | | ) ) )\ ) ( /(| / ( )) ) ) )) ) ( ( _(|)_) ) ( ||\(|(|)|/|| ) (|(||(||)) ( //|/l|||)|\\ \ ) (/ / // /|//\\ \ \ \ _) --- Node: 9889a7c79e9b --- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 367, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 118, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 167, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1017, in clear include_upstream=upstream) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870, in sub_dag dag = copy.deepcopy(self) File "/usr/lib/python2.7/copy.py", line 174, in deepcopy y = copier(memo) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/lib/python2.7/copy.py", line 174, in deepcopy y = copier(memo) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/lib/python2.7/copy.py", line 163, in
[jira] [Closed] (AIRFLOW-344) ImportError: No module named lockfile.pidlockfile
[
https://issues.apache.org/jira/browse/AIRFLOW-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-344.
---
Resolution: Auto Closed
> ImportError: No module named lockfile.pidlockfile
> -
>
> Key: AIRFLOW-344
> URL: https://issues.apache.org/jira/browse/AIRFLOW-344
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Greg Reda
>Priority: Major
>
> When setting up a clean install of airflow on ubuntu 14.04, I ran into the
> following error:
> {code}
> vagrant@vagrant-ubuntu-trusty-64:~$ airflow
> [2016-07-19 15:37:41,839] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-07-19 15:37:41,912] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-07-19 15:37:41,929] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 5, in
> from airflow.bin.cli import CLIFactory
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 17,
> in
> from daemon.pidfile import TimeoutPIDLockFile
> File "/usr/local/lib/python2.7/dist-packages/daemon/pidfile.py", line 18,
> in
> from lockfile.pidlockfile import PIDLockFile
> ImportError: No module named lockfile.pidlockfile
> {code}
> This seems to be because Airflow includes python-daemon >= 2.1.1 as a
> dependency, but not lockfile. It seems that some time before 2.1.1
> python-daemon removed TimeoutPIDLockFile and instead decided just to use
> lockfile.
> Uninstalling python-daemon and reinstalling fixed the issue, as python-daemon
> included a lockfile dependency.
> Not sure why PyPi wouldn't have pulled this down when resolving Airflow's
> python-daemon dependency, but figured I'd open this issue in case other run
> into it in the future. It might make sense for Airflow to explicitly require
> lockfile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-332) Documentation says Sub-DAG names follow a convention, but in reality it's enforced
[
https://issues.apache.org/jira/browse/AIRFLOW-332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-332.
---
Resolution: Auto Closed
> Documentation says Sub-DAG names follow a convention, but in reality it's
> enforced
> --
>
> Key: AIRFLOW-332
> URL: https://issues.apache.org/jira/browse/AIRFLOW-332
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Reporter: Jon McKenzie
>Priority: Minor
>
> Currently, the documentation states that a Sub-DAG's name should follow the
> convention {{.}}. In reality, this is not just a
> convention, but it is enforced, and a {{SubDagOperator}} will fail to load if
> its name is incorrect. The documentation should be updated to be a little
> clearer in this regard.
> Ideally, it would also be nice to either specify via an option that this
> validation should take place (or not), or be easier to subclass
> {{SubDagOperator}} and override the validation, so that users can have their
> own naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-332) Documentation says Sub-DAG names follow a convention, but in reality it's enforced
[
https://issues.apache.org/jira/browse/AIRFLOW-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069115#comment-17069115
]
Daniel Imberman commented on AIRFLOW-332:
-
This issue has been moved to https://github.com/apache/airflow/issues/7928
> Documentation says Sub-DAG names follow a convention, but in reality it's
> enforced
> --
>
> Key: AIRFLOW-332
> URL: https://issues.apache.org/jira/browse/AIRFLOW-332
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Reporter: Jon McKenzie
>Priority: Minor
>
> Currently, the documentation states that a Sub-DAG's name should follow the
> convention {{.}}. In reality, this is not just a
> convention, but it is enforced, and a {{SubDagOperator}} will fail to load if
> its name is incorrect. The documentation should be updated to be a little
> clearer in this regard.
> Ideally, it would also be nice to either specify via an option that this
> validation should take place (or not), or be easier to subclass
> {{SubDagOperator}} and override the validation, so that users can have their
> own naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7928: Documentation says Sub-DAG names follow a convention, but in reality it's enforced
dimberman opened a new issue #7928: Documentation says Sub-DAG names follow a convention, but in reality it's enforced URL: https://github.com/apache/airflow/issues/7928 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: Currently, the documentation states that a Sub-DAG's name should follow the convention .. In reality, this is not just a convention, but it is enforced, and a SubDagOperator will fail to load if its name is incorrect. The documentation should be updated to be a little clearer in this regard. Ideally, it would also be nice to either specify via an option that this validation should take place (or not), or be easier to subclass SubDagOperator and override the validation, so that users can have their own naming conventions. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-326. --- Resolution: Auto Closed > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069114#comment-17069114 ] Daniel Imberman commented on AIRFLOW-326: - This issue has been moved to https://github.com/apache/airflow/issues/7927 > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-330) Decorated PythonOperator python_callable functions don't show the original function in task code view
[
https://issues.apache.org/jira/browse/AIRFLOW-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-330.
---
Resolution: Auto Closed
> Decorated PythonOperator python_callable functions don't show the original
> function in task code view
> -
>
> Key: AIRFLOW-330
> URL: https://issues.apache.org/jira/browse/AIRFLOW-330
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Reporter: Jon McKenzie
>Priority: Minor
>
> In Python 3.4 or below, if you try to decorate the {{python_callable}} to a
> {{PythonOperator}} in the following manner (i.e. like the manual application
> of a standard Python decorator using {{functools.wraps}}):
> {noformat}
> task.python_callable = wrap(task.python_callable)
> {noformat}
> ...the code view of that task in the web UI shows the code for the {{wrap}}
> function rather than the initial {{python_callable}}.
> The fix is to run something like this (where {{inspect.unwrap}} is available
> in Python 3.4+):
> {noformat}
> inspect.getsource(inspect.unwrap(func))
> {noformat}
> ...rather than:
> {noformat}
> inspect.getsource(func)
> {noformat}
> I'm not sure if this is something worth fixing or not, since I believe Python
> 3.5+ implements the above fix (although I believe it would still be an issue
> in Python 2.x).
> Just for some background, I'm writing a higher level API around Airflow that
> takes tasks as arguments and connects their inputs via {{XCom}} (among other
> things). The callables I want my API users to write aren't going to need
> access to any of the task context (only so that they don't need to know
> Airflow internals), hence the need to decorate them appropriately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7927: Unclean scheduler restarts result in zombie Task Instances
dimberman opened a new issue #7927: Unclean scheduler restarts result in zombie Task Instances URL: https://github.com/apache/airflow/issues/7927 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: When a task is running and the scheduler and task both die before the task completes execution, the task is not identified as failed. The task appears to be in a "Running" state even though it has long since stopped running. **What you expected to happen**: **How to reproduce it**: **Anything else we need to know**: Moved here from https://issues.apache.org/jira/browse/AIRFLOW-326 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Issue Comment Deleted] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman updated AIRFLOW-326: Comment: was deleted (was: This issue has been moved to https://github.com/apache/airflow/issues/7926) > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman reopened AIRFLOW-326: - > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069113#comment-17069113 ] Daniel Imberman commented on AIRFLOW-326: - This issue has been moved to https://github.com/apache/airflow/issues/7926 > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances
[ https://issues.apache.org/jira/browse/AIRFLOW-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-326. --- Resolution: Auto Closed > Unclean scheduler restarts result in zombie Task Instances > -- > > Key: AIRFLOW-326 > URL: https://issues.apache.org/jira/browse/AIRFLOW-326 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.7.1.3 >Reporter: George Leslie-Waksman >Priority: Major > > When a task is running and the scheduler and task both die before the task > completes execution, the task is not identified as failed. The task appears > to be in a "Running" state even though it has long since stopped running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-108) Add data retention policy to Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-108. --- Resolution: Auto Closed > Add data retention policy to Airflow > > > Key: AIRFLOW-108 > URL: https://issues.apache.org/jira/browse/AIRFLOW-108 > Project: Apache Airflow > Issue Type: Wish > Components: database >Reporter: Chris Riccomini >Priority: Major > > Airflow's DB currently holds the entire history of all executions for all > time. This is problematic as the DB grows. The UI starts to get slower, and > the DB's disk usage grows. There is no bound to how large the DB will grow. > It would be useful to add a feature in Airflow to do two things: > # Delete old data from the DB > # Mark some lower watermark, past which DAG executions are ignored > For example, (2) would allow you to tell the scheduler "ignore all data prior > to a year ago". And (1) would allow Airflow to delete all data prior to > January 1, 2015. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] KevinYang21 commented on issue #7913: [AIRFLOW-7113] fix gantt render error
KevinYang21 commented on issue #7913: [AIRFLOW-7113] fix gantt render error URL: https://github.com/apache/airflow/pull/7913#issuecomment-605352958 The CI error looks irrelevant tho, maybe try rebase and rerun? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman opened a new issue #7926: Kill task instances that haven't been able to heartbeat for a while
dimberman opened a new issue #7926: Kill task instances that haven't been able to heartbeat for a while URL: https://github.com/apache/airflow/issues/7926 **Description** A task run by the LocalTaskJob periodically updates a timestamp to indicate that the task is still alive and running. If the task is unable to update this timestamp for a long time (for example, due to DB connection errors), the scheduler may reschedule the task to run again. In such a case, it's possible that two instances of the task are running. The task can monitor the time since last heartbeat and kill itself to prevent such cases. **Use case / motivation** **Related Issues** Moved here from https://issues.apache.org/jira/browse/AIRFLOW-374 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KevinYang21 commented on issue #7913: [AIRFLOW-7113] fix gantt render error
KevinYang21 commented on issue #7913: [AIRFLOW-7113] fix gantt render error URL: https://github.com/apache/airflow/pull/7913#issuecomment-605352573 @kaxil I think if we have tasks marked as failed by the scheduler upon receiving executor event then we'll end up having such failing tasks w/o a start date and crash the gantt chart. @cong-zhu correct me if I'm wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-374) Kill task instances that haven't been able to heartbeat for a while
[ https://issues.apache.org/jira/browse/AIRFLOW-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069107#comment-17069107 ] Daniel Imberman commented on AIRFLOW-374: - This issue has been moved to https://github.com/apache/airflow/issues/7925 > Kill task instances that haven't been able to heartbeat for a while > --- > > Key: AIRFLOW-374 > URL: https://issues.apache.org/jira/browse/AIRFLOW-374 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Paul Yang >Assignee: Paul Yang >Priority: Major > Fix For: 1.8.0 > > > A task run by the LocalTaskJob periodically updates a timestamp to indicate > that the task is still alive and running. If the task is unable to update > this timestamp for a long time (for example, due to DB connection errors), > the scheduler may reschedule the task to run again. In such a case, it's > possible that two instances of the task are running. The task can monitor the > time since last heartbeat and kill itself to prevent such cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-374) Kill task instances that haven't been able to heartbeat for a while
[ https://issues.apache.org/jira/browse/AIRFLOW-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-374. --- Resolution: Auto Closed > Kill task instances that haven't been able to heartbeat for a while > --- > > Key: AIRFLOW-374 > URL: https://issues.apache.org/jira/browse/AIRFLOW-374 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Paul Yang >Assignee: Paul Yang >Priority: Major > Fix For: 1.8.0 > > > A task run by the LocalTaskJob periodically updates a timestamp to indicate > that the task is still alive and running. If the task is unable to update > this timestamp for a long time (for example, due to DB connection errors), > the scheduler may reschedule the task to run again. In such a case, it's > possible that two instances of the task are running. The task can monitor the > time since last heartbeat and kill itself to prevent such cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman opened a new issue #7925: Kill task instances that haven't been able to heartbeat for a while
dimberman opened a new issue #7925: Kill task instances that haven't been able to heartbeat for a while URL: https://github.com/apache/airflow/issues/7925 **Description** A task run by the LocalTaskJob periodically updates a timestamp to indicate that the task is still alive and running. If the task is unable to update this timestamp for a long time (for example, due to DB connection errors), the scheduler may reschedule the task to run again. In such a case, it's possible that two instances of the task are running. The task can monitor the time since last heartbeat and kill itself to prevent such cases. **Use case / motivation** **Related Issues** Moved here from https://issues.apache.org/jira/browse/AIRFLOW-374 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
