[GitHub] [airflow] potiuk merged pull request #7207: [AIRFLOW-XXXX] Consistency fixes in new documentation
potiuk merged pull request #7207: [AIRFLOW-] Consistency fixes in new documentation URL: https://github.com/apache/airflow/pull/7207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6636) Sometimes start/execution/end dates are not set for tasks (mostly in tests)
[ https://issues.apache.org/jira/browse/AIRFLOW-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6636. --- Fix Version/s: 1.10.8 Resolution: Fixed > Sometimes start/execution/end dates are not set for tasks (mostly in tests) > --- > > Key: AIRFLOW-6636 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6636 > Project: Apache Airflow > Issue Type: Bug > Components: ci, tests >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > > Sometimes (mainly in tests) tasks do not have start/execution/end date set > and printing task instance information throws an exception which cover the > real problems and makes diagnostics difficult. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6636) Sometimes start/execution/end dates are not set for tasks (mostly in tests)
[ https://issues.apache.org/jira/browse/AIRFLOW-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028307#comment-17028307 ] ASF subversion and git services commented on AIRFLOW-6636: -- Commit 4d6198c19fda61142618fdb5c327ced5266dc937 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4d6198c ] [AIRFLOW-6636] Avoid exceptions when printing task instance (#7257) Sometimes (mainly in tests) tasks do not have start/execution/end date set and printing task instance information throws an exception which cover the real problems and makes diagnostics difficult. > Sometimes start/execution/end dates are not set for tasks (mostly in tests) > --- > > Key: AIRFLOW-6636 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6636 > Project: Apache Airflow > Issue Type: Bug > Components: ci, tests >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > > Sometimes (mainly in tests) tasks do not have start/execution/end date set > and printing task instance information throws an exception which cover the > real problems and makes diagnostics difficult. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6636) Sometimes start/execution/end dates are not set for tasks (mostly in tests)
[ https://issues.apache.org/jira/browse/AIRFLOW-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028306#comment-17028306 ] ASF GitHub Bot commented on AIRFLOW-6636: - potiuk commented on pull request #7257: [AIRFLOW-6636] Avoid exceptions when printing task instance URL: https://github.com/apache/airflow/pull/7257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Sometimes start/execution/end dates are not set for tasks (mostly in tests) > --- > > Key: AIRFLOW-6636 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6636 > Project: Apache Airflow > Issue Type: Bug > Components: ci, tests >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > > Sometimes (mainly in tests) tasks do not have start/execution/end date set > and printing task instance information throws an exception which cover the > real problems and makes diagnostics difficult. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7257: [AIRFLOW-6636] Avoid exceptions when printing task instance
potiuk merged pull request #7257: [AIRFLOW-6636] Avoid exceptions when printing task instance URL: https://github.com/apache/airflow/pull/7257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6710) Lazy create custom executor modules
[ https://issues.apache.org/jira/browse/AIRFLOW-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028305#comment-17028305 ] ASF GitHub Bot commented on AIRFLOW-6710: - mik-laj commented on pull request #7333: [AIRFLOW-6710] Lazy create custom executor modules URL: https://github.com/apache/airflow/pull/7333 Currently, modules for the executor are initialized when loading the airflow module and when calling the `ExecutorLoader.get_default_executor` method. I think the second case is sufficient. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lazy create custom executor modules > --- > > Key: AIRFLOW-6710 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6710 > Project: Apache Airflow > Issue Type: Improvement > Components: executors, plugins >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7333: [AIRFLOW-6710] Lazy create custom executor modules
mik-laj opened a new pull request #7333: [AIRFLOW-6710] Lazy create custom executor modules URL: https://github.com/apache/airflow/pull/7333 Currently, modules for the executor are initialized when loading the airflow module and when calling the `ExecutorLoader.get_default_executor` method. I think the second case is sufficient. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6710) Lazy create custom executor modules
[ https://issues.apache.org/jira/browse/AIRFLOW-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-6710: --- Summary: Lazy create custom executor modules (was: Lazy initialise executor modules) > Lazy create custom executor modules > --- > > Key: AIRFLOW-6710 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6710 > Project: Apache Airflow > Issue Type: Improvement > Components: executors, plugins >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6710) Lazy initialise executor modules
Kamil Bregula created AIRFLOW-6710: -- Summary: Lazy initialise executor modules Key: AIRFLOW-6710 URL: https://issues.apache.org/jira/browse/AIRFLOW-6710 Project: Apache Airflow Issue Type: Improvement Components: executors, plugins Affects Versions: 1.10.7 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk closed pull request #7302: [AIRFLOW-6690] Fix number of logs to check in MySQL
potiuk closed pull request #7302: [AIRFLOW-6690] Fix number of logs to check in MySQL URL: https://github.com/apache/airflow/pull/7302 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6690) MYSQL after long run can restart and have more start logs
[ https://issues.apache.org/jira/browse/AIRFLOW-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028302#comment-17028302 ] ASF GitHub Bot commented on AIRFLOW-6690: - potiuk commented on pull request #7302: [AIRFLOW-6690] Fix number of logs to check in MySQL URL: https://github.com/apache/airflow/pull/7302 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MYSQL after long run can restart and have more start logs > - > > Key: AIRFLOW-6690 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6690 > Project: Apache Airflow > Issue Type: Bug > Components: breeze >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6709) Moving example_dags in git_sync caused master failure
[ https://issues.apache.org/jira/browse/AIRFLOW-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6709. --- Fix Version/s: 2.0.0 Resolution: Fixed > Moving example_dags in git_sync caused master failure > - > > Key: AIRFLOW-6709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6709 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6242) Introduce run_type in DAG Run table
[ https://issues.apache.org/jira/browse/AIRFLOW-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028300#comment-17028300 ] Kamil Bregula commented on AIRFLOW-6242: We have one more prefix - "manual": [https://github.com/apache/airflow/blob/41a9bfd/airflow/api/common/experimental/trigger_dag.py#L72] > Introduce run_type in DAG Run table > --- > > Key: AIRFLOW-6242 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6242 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Hello, > Before you start reading this ticket, I want to tell you one thing - you are > wonderful. > We use queries that use like expressions to filter two types of DAG Run. > * Backfill: > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/jobs/backfill_job.py#L52] > * Scheduled > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/models/dagrun.py#L46] > This type of expression can affect performance and it seems a better idea to > store this information in a separate column. This may allow a query planner > to create a better query plan in the database engine. > Thanks for reading this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7332: [AIRFLOW-6709] Fixed failing git sync
potiuk opened a new pull request #7332: [AIRFLOW-6709] Fixed failing git sync URL: https://github.com/apache/airflow/pull/7332 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6709) Moving example_dags in git_sync caused master failure
[ https://issues.apache.org/jira/browse/AIRFLOW-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028296#comment-17028296 ] ASF GitHub Bot commented on AIRFLOW-6709: - potiuk commented on pull request #7332: [AIRFLOW-6709] Fixed failing git sync URL: https://github.com/apache/airflow/pull/7332 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Moving example_dags in git_sync caused master failure > - > > Key: AIRFLOW-6709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6709 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7332: [AIRFLOW-6709] Fixed failing git sync
potiuk commented on issue #7332: [AIRFLOW-6709] Fixed failing git sync URL: https://github.com/apache/airflow/pull/7332#issuecomment-581106263 I am merging it quickly to fix master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7332: [AIRFLOW-6709] Fixed failing git sync
potiuk merged pull request #7332: [AIRFLOW-6709] Fixed failing git sync URL: https://github.com/apache/airflow/pull/7332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6709) Moving example_dags in git_sync caused master failure
[ https://issues.apache.org/jira/browse/AIRFLOW-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028297#comment-17028297 ] ASF GitHub Bot commented on AIRFLOW-6709: - potiuk commented on pull request #7332: [AIRFLOW-6709] Fixed failing git sync URL: https://github.com/apache/airflow/pull/7332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Moving example_dags in git_sync caused master failure > - > > Key: AIRFLOW-6709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6709 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6709) Moving example_dags in git_sync caused master failure
Jarek Potiuk created AIRFLOW-6709: - Summary: Moving example_dags in git_sync caused master failure Key: AIRFLOW-6709 URL: https://issues.apache.org/jira/browse/AIRFLOW-6709 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator
codecov-io edited a comment on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator URL: https://github.com/apache/airflow/pull/7276#issuecomment-581100536 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=h1) Report > Merging [#7276](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `96.66%`. [](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7276 +/- ## == + Coverage85.6% 85.83% +0.23% == Files 863 863 Lines 4052040548 +28 == + Hits3468634806 +120 + Misses 5834 5742 -92 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `94.72% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.58% <100%> (+0.06%)` | :arrow_up: | | [airflow/models/skipmixin.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2tpcG1peGluLnB5) | `97.67% <100%> (+0.11%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `95.39% <100%> (+0.25%)` | :arrow_up: | | [airflow/operators/branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYnJhbmNoX29wZXJhdG9yLnB5) | `86.66% <87.5%> (-0.84%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=footer). Last update [39375c0...2c7fae0](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7030: [AIRFLOW-6440][AIP-29] Add AWS Fargate Executor
codecov-io commented on issue #7030: [AIRFLOW-6440][AIP-29] Add AWS Fargate Executor URL: https://github.com/apache/airflow/pull/7030#issuecomment-581103197 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=h1) Report > Merging [#7030](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.26%`. > The diff coverage is `92.82%`. [](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7030 +/- ## == + Coverage85.6% 85.86% +0.26% == Files 863 865 +2 Lines 4052040743 +223 == + Hits3468634986 +300 + Misses 5834 5757 -77 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/executor\_loader.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZXhlY3V0b3JfbG9hZGVyLnB5) | `79.48% <100%> (+0.53%)` | :arrow_up: | | [airflow/config\_templates/default\_fargate.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfZmFyZ2F0ZS5weQ==) | `85.71% <85.71%> (ø)` | | | [airflow/executors/fargate\_executor.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZmFyZ2F0ZV9leGVjdXRvci5weQ==) | `93.02% <93.02%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `77.4% <0%> (+0.96%)` | :arrow_up: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `100% <0%> (+45%)` | :arrow_up: | | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=footer). Last update [39375c0...f1c6c04](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yuqian90 commented on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator
yuqian90 commented on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator URL: https://github.com/apache/airflow/pull/7276#issuecomment-581103041 - Added tests - Extended the same support to ``BranchPythonOperator``, ``ShortCircuitOperator`` and ``BaseBranchOperator``. Clearing downstream tasks of all these three operators will now do the right thing. - That said, I still think ``BaseBranchOperator`` is a bit redundant since we already have ``BranchPythonOperator``. I'm proposing to deprecate ``BaesBranchOperator`` in [another PR](https://github.com/apache/airflow/pull/7331) The PR is now ready for review. Reviewers @kaxil @ashb @feluelle please take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator
codecov-io edited a comment on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator URL: https://github.com/apache/airflow/pull/7276#issuecomment-581100536 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=h1) Report > Merging [#7276](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.22%`. > The diff coverage is `76.66%`. [](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7276 +/- ## == + Coverage 85.59% 85.82% +0.22% == Files 863 863 Lines 4052040548 +28 == + Hits3468534800 +115 + Misses 5835 5748 -87 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `94.72% <100%> (ø)` | :arrow_up: | | [airflow/models/skipmixin.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2tpcG1peGluLnB5) | `97.67% <100%> (+0.11%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `92.1% <44.44%> (-3.04%)` | :arrow_down: | | [airflow/operators/branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYnJhbmNoX29wZXJhdG9yLnB5) | `86.66% <87.5%> (-0.84%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.38% <90%> (-0.14%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.88% <0%> (+0.28%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `77.4% <0%> (+0.96%)` | :arrow_up: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=footer). Last update [39375c0...9a907c9](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator
codecov-io commented on issue #7276: [AIRFLOW-5391] Make BaseOperator respect branching of parent BranchPythonOperator URL: https://github.com/apache/airflow/pull/7276#issuecomment-581100536 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=h1) Report > Merging [#7276](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.22%`. > The diff coverage is `76.66%`. [](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7276 +/- ## == + Coverage 85.59% 85.82% +0.22% == Files 863 863 Lines 4052040548 +28 == + Hits3468534800 +115 + Misses 5835 5748 -87 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `94.72% <100%> (ø)` | :arrow_up: | | [airflow/models/skipmixin.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2tpcG1peGluLnB5) | `97.67% <100%> (+0.11%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `92.1% <44.44%> (-3.04%)` | :arrow_down: | | [airflow/operators/branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYnJhbmNoX29wZXJhdG9yLnB5) | `86.66% <87.5%> (-0.84%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.38% <90%> (-0.14%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.88% <0%> (+0.28%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `77.4% <0%> (+0.96%)` | :arrow_up: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/7276/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=footer). Last update [39375c0...9a907c9](https://codecov.io/gh/apache/airflow/pull/7276?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config
codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249#issuecomment-577972968 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=h1) Report > Merging [#7249](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr=desc) will **increase** coverage by `0.36%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7249 +/- ## = + Coverage 85.34% 85.7% +0.36% = Files 791 863 +72 Lines 40124 40520 +396 = + Hits34243 34729 +486 + Misses 58815791 -90 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <ø> (+0.23%)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | ... and [466 more](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=footer). Last update [21a4651...e29bead](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config
codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249#issuecomment-577972968 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=h1) Report > Merging [#7249](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr=desc) will **increase** coverage by `0.48%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7249 +/- ## == + Coverage 85.34% 85.83% +0.48% == Files 791 863 +72 Lines 4012440520 +396 == + Hits3424334779 +536 + Misses 5881 5741 -140 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <ø> (+0.23%)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | ... and [463 more](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=footer). Last update [21a4651...e29bead](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config
codecov-io edited a comment on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249#issuecomment-577972968 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=h1) Report > Merging [#7249](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr=desc) will **increase** coverage by `0.36%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7249 +/- ## = + Coverage 85.34% 85.7% +0.36% = Files 791 863 +72 Lines 40124 40520 +396 = + Hits34243 34729 +486 + Misses 58815791 -90 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <ø> (+0.23%)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | ... and [466 more](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=footer). Last update [21a4651...e29bead](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] xinbinhuang commented on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config
xinbinhuang commented on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249#issuecomment-581098112 @mik-laj @kaxil Hi, I refactored the tests and stop trying to use `@conf_vars` to test the change on default_config. The reason is that when I try to do `importlib.reload(baseoperator)` to refresh the config, it will also mess up with the test on `TestBaseOperatorMethods::test_cross_downstream` and `TestBaseOperatorMethods::test_chain`. After the reload, it leads to the weird behavior below and I am not sure what's going on. ``` # This can be reproduced by adding below after the imports in `test_baseoperator.py` ### import importlib ### from airflow.models import baseoperator ### importlib.reload(baseoperator) # It will fails on the below checks in BaseOperator.chain() and BaseOperator._set_relative isinstance(task, BaseOperator) # here task = DummyOperator() False ``` I would love to fix this if there are more pointers, but I believe right now the test has already served the purpose of testing the default values are boolean. PTAL and let me know how you think. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7324: [AIRFLOW-6704] Set TaskInstance.operator in constructor
codecov-io commented on issue #7324: [AIRFLOW-6704] Set TaskInstance.operator in constructor URL: https://github.com/apache/airflow/pull/7324#issuecomment-581094712 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=h1) Report > Merging [#7324](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7324 +/- ## == + Coverage 85.59% 85.83% +0.23% == Files 863 863 Lines 4052040521 +1 == + Hits3468534780 +95 + Misses 5835 5741 -94 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `94.72% <100%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.88% <0%> (+0.28%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `77.4% <0%> (+0.96%)` | :arrow_up: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `100% <0%> (+45%)` | :arrow_up: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (+45.65%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7324/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=footer). Last update [39375c0...273e79b](https://codecov.io/gh/apache/airflow/pull/7324?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yuqian90 opened a new pull request #7331: [AIRFLOW-6657] Deprecate branch operator
yuqian90 opened a new pull request #7331: [AIRFLOW-6657] Deprecate branch operator URL: https://github.com/apache/airflow/pull/7331 Everything that can be done by deriving from ``BaseBranchOperator`` can also be done using ``BranchPythonOperator`` with more concise and readable code. E.g. ``LatestOnlyOperator`` can be implemented from ``BranchPythonOperator`` easily. The test tasks in ``test_branch_operator.py`` are also better implemented using ``BranchPythonOperator`` like the ones in ``tests/operators/test_python.py`` The changes in this PR to ``LatestOnlyOperator`` does not require additional testing because there's already a ``test_latest_only_operator.py`` that works fine after the change. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6657) Deprecate BaseBranchOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028259#comment-17028259
]
ASF GitHub Bot commented on AIRFLOW-6657:
-
yuqian90 commented on pull request #7331: [AIRFLOW-6657] Deprecate branch
operator
URL: https://github.com/apache/airflow/pull/7331
Everything that can be done by deriving from ``BaseBranchOperator`` can also
be done using ``BranchPythonOperator`` with more concise and readable code.
E.g. ``LatestOnlyOperator`` can be implemented from ``BranchPythonOperator``
easily. The test tasks in ``test_branch_operator.py`` are also better
implemented using ``BranchPythonOperator`` like the ones in
``tests/operators/test_python.py``
The changes in this PR to ``LatestOnlyOperator`` does not require additional
testing because there's already a ``test_latest_only_operator.py`` that works
fine after the change.
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Deprecate BaseBranchOperator
>
>
> Key: AIRFLOW-6657
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6657
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.4
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
>
> This PR added a base class {{BaseBranchOperator}} that is intended to let
> people implement custom branching logic as its subclasses.
> https://github.com/apache/airflow/pull/5231
> However the existing {{BranchPythonOperator}} is implemented on its own, not
> utilizing {{BaseBranchOperator}}.
> This is making other JIRAs difficult to continue. E.g. this one AIRFLOW-5391.
> Should subsequent JIRAs be working on {{BranchPythonOperator}} or
> {{BaseBranchOperator}} or both?
> Everything that can be done by deriving from {{BaseBranchOperator}} can also
> be done by deriving from {{BranchPythonOperator}}, often with more readable
> code.
> For example, the only current internal use of {{BaseBranchOperator}} is in
> {{LatestOnlyOperator}}. The code can be changed to something like this
> without any loss of flexibility. In fact the new code may look more readable
> because it has no dictionary look-ups from {{context}}:
> {code:python}
> from airflow.operators.python import BranchPythonOperator
> class LatestOnlyOperator(BranchPythonOperator):
> """
> Allows a workflow to skip tasks that are not running during the most
> recent schedule interval.
> If the task is run outside of the latest schedule interval (i.e.
> external_trigger),
> all directly downstream tasks will be skipped.
> Note that downstream tasks are never skipped if the given DAG_Run is
> marked as externally triggered.
> """
> ui_color = '#e9ffdb' # nyanza
> def __init__(self, *args, **kwargs):
> def python_callable(dag_run, task, dag, execution_date, **_):
> # If the DAG Run is externally triggered, then return without
> # skipping downstream tasks
> if dag_run and dag_run.external_trigger:
> self.log.info(
>
[jira] [Commented] (AIRFLOW-6208) StreamLogWriter class does not implement file fileno() function, preventing output redirection to stdout with the python subprocess module
[
https://issues.apache.org/jira/browse/AIRFLOW-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028247#comment-17028247
]
ASF GitHub Bot commented on AIRFLOW-6208:
-
stale[bot] commented on pull request #6767: [AIRFLOW-6208] Implement fileno in
StreamLogWriter
URL: https://github.com/apache/airflow/pull/6767
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> StreamLogWriter class does not implement file fileno() function, preventing
> output redirection to stdout with the python subprocess module
> --
>
> Key: AIRFLOW-6208
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6208
> Project: Apache Airflow
> Issue Type: Bug
> Components: breeze, logging
>Affects Versions: 1.10.6
> Environment: Archlinux python 3.8 Linux 5.4
> Also affects airflow breeze python 3.6 docker development environment
>Reporter: Sahas L
>Assignee: Sahas L
>Priority: Major
>
> When using the python operator to execute a callbackfunction which makes use
> of the subprocess module stdout redirection to system stdout, the task fails
> due to fileno() being unimplemented in airflow/utils/log/logging_mixin.py
> class StreamLogWriter.
> {code:java}
> Traceback (most recent call last):
> File "/usr/lib/python3.8/site-packages/airflow/models/taskinstance.py",
> line 930, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/lib/python3.8/site-packages/airflow/operators/python_operator.py", line
> 113, in execute
> return_value = self.execute_callable()
> File
> "/usr/lib/python3.8/site-packages/airflow/operators/python_operator.py", line
> 118, in execute_callable
> return self.python_callable(*self.op_args, **self.op_kwargs)
> File "/home/sahas/airflow/dags/subprocess.py", line 39, in sp
> subprocess.run("echo 1", shell=True, check=True, stdout=sys.stdout,
> stderr=sys.stderr)
> File "/usr/lib/python3.8/subprocess.py", line 489, in run
> with Popen(*popenargs, **kwargs) as process:
> File "/usr/lib/python3.8/subprocess.py", line 804, in __init__
> errread, errwrite) = self._get_handles(stdin, stdout, stderr)
> File "/usr/lib/python3.8/subprocess.py", line 1487, in _get_handles
> c2pwrite = stdout.fileno()
> AttributeError: 'StreamLogWriter' object has no attribute 'fileno'
> {code}
> The code in the callback function is:
> {code:java}
> import sys
> import os
> subprocess.run("echo 1", shell=True, check=True, stdout=sys.stdout,
> stderr=sys.stderr)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #6767: [AIRFLOW-6208] Implement fileno in StreamLogWriter
stale[bot] closed pull request #6767: [AIRFLOW-6208] Implement fileno in StreamLogWriter URL: https://github.com/apache/airflow/pull/6767 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6014) Kubernetes executor - handle preempted deleted pods - queued tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028246#comment-17028246 ] ASF GitHub Bot commented on AIRFLOW-6014: - stale[bot] commented on pull request #6606: [AIRFLOW-6014] - handle pods which are preempted and deleted by kuber… URL: https://github.com/apache/airflow/pull/6606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kubernetes executor - handle preempted deleted pods - queued tasks > -- > > Key: AIRFLOW-6014 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6014 > Project: Apache Airflow > Issue Type: Improvement > Components: executor-kubernetes >Affects Versions: 1.10.6 >Reporter: afusr >Assignee: Daniel Imberman >Priority: Minor > > We have encountered an issue whereby when using the kubernetes executor, and > using autoscaling, airflow pods are preempted and airflow never attempts to > rerun these pods. > This is partly as a result of having the following set on the pod spec: > restartPolicy: Never > This makes sense as if a pod fails when running a task, we don't want > kubernetes to retry it, as this should be controlled by airflow. > What we believe happens is that when a new node is added by autoscaling, > kubernetes schedules a number of airflow pods onto the new node, as well as > any pods required by k8s/daemon sets. As these are higher priority, the > Airflow pods are preempted, and deleted. You see messages such as: > > Preempted by kube-system/ip-masq-agent-xz77q on node > gke-some--airflow--node-1ltl > > Within the kubernetes executor, these pods end up in a status of pending and > an event of deleted is received but not handled. > The end result is tasks remain in a queued state forever. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #6606: [AIRFLOW-6014] - handle pods which are preempted and deleted by kuber…
stale[bot] closed pull request #6606: [AIRFLOW-6014] - handle pods which are preempted and deleted by kuber… URL: https://github.com/apache/airflow/pull/6606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581067534 > This way we could avoid this lazy-loading for specific imports and have it also solved for all future cases We want to load lazily because some users do not want to use certain operators, so their code should be loaded when it is needed. Now all classes are loaded for each user. When we will move this method call, it will still load all classes for all users, not according to the user's needs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6708) Set unique logger names
[ https://issues.apache.org/jira/browse/AIRFLOW-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028224#comment-17028224 ] ASF GitHub Bot commented on AIRFLOW-6708: - mik-laj commented on pull request #7330: [AIRFLOW-6708] Set unique logger names URL: https://github.com/apache/airflow/pull/7330 LoggingMixin is intended for configuring the logger for classes. If it is used incorrectly then all loggers are configured for one name - `airflow.utils.log.logging_mixin.LoggingMixin` Before: ``` $ AIRFLOW__LOGGING__COLORED_LOG_FORMAT='%(name)s: %(message)s' python -c 'from airflow.logging_config import configure_logging; from airflow.executors.celery_executor import execute_command; configure_logging(); execute_command(["bash", "-c", "echo 1"]);' airflow.utils.log.logging_mixin.LoggingMixin: Executing command in Celery: ['bash', '-c', 'echo 1'] 1 ``` After: ``` 23:33 $ AIRFLOW__LOGGING__COLORED_LOG_FORMAT='%(name)s: %(message)s' python -c 'from airflow.logging_config import configure_logging; from airflow.executors.celery_executor import execute_command; configure_logging(); execute_command(["bash", "-c", "echo 1"]);' airflow.executors.celery_executor: Executing command in Celery: ['bash', '-c', 'echo 1'] ` ``` --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Set unique logger names > --- > > Key: AIRFLOW-6708 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6708 > Project: Apache Airflow > Issue Type: Improvement > Components: core, logging >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7330: [AIRFLOW-6708] Set unique logger names
mik-laj opened a new pull request #7330: [AIRFLOW-6708] Set unique logger names URL: https://github.com/apache/airflow/pull/7330 LoggingMixin is intended for configuring the logger for classes. If it is used incorrectly then all loggers are configured for one name - `airflow.utils.log.logging_mixin.LoggingMixin` Before: ``` $ AIRFLOW__LOGGING__COLORED_LOG_FORMAT='%(name)s: %(message)s' python -c 'from airflow.logging_config import configure_logging; from airflow.executors.celery_executor import execute_command; configure_logging(); execute_command(["bash", "-c", "echo 1"]);' airflow.utils.log.logging_mixin.LoggingMixin: Executing command in Celery: ['bash', '-c', 'echo 1'] 1 ``` After: ``` 23:33 $ AIRFLOW__LOGGING__COLORED_LOG_FORMAT='%(name)s: %(message)s' python -c 'from airflow.logging_config import configure_logging; from airflow.executors.celery_executor import execute_command; configure_logging(); execute_command(["bash", "-c", "echo 1"]);' airflow.executors.celery_executor: Executing command in Celery: ['bash', '-c', 'echo 1'] ` ``` --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter
codecov-io edited a comment on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter URL: https://github.com/apache/airflow/pull/7329#issuecomment-581075478 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=h1) Report > Merging [#7329](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7329 +/- ## == + Coverage 85.59% 85.83% +0.23% == Files 863 863 Lines 4052040520 == + Hits3468534779 +94 + Misses 5835 5741 -94 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <100%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.88% <0%> (+0.28%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `77.4% <0%> (+0.96%)` | :arrow_up: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `100% <0%> (+45%)` | :arrow_up: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (+45.65%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=footer). Last update [39375c0...7c34ff1](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter
codecov-io commented on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter URL: https://github.com/apache/airflow/pull/7329#issuecomment-581075478 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=h1) Report > Merging [#7329](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/39375c01ced008a715bb2a540df31fe2cf908bbb?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7329 +/- ## == + Coverage 85.59% 85.83% +0.23% == Files 863 863 Lines 4052040520 == + Hits3468534779 +94 + Misses 5835 5741 -94 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <100%> (ø)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7329/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.88% <0%> (+0.28%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=footer). Last update [39375c0...7c34ff1](https://codecov.io/gh/apache/airflow/pull/7329?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6708) Set unique logger names
Kamil Bregula created AIRFLOW-6708: -- Summary: Set unique logger names Key: AIRFLOW-6708 URL: https://issues.apache.org/jira/browse/AIRFLOW-6708 Project: Apache Airflow Issue Type: Improvement Components: core, logging Affects Versions: 1.10.7 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3349) StreamLogWriter encoding is wrong type
[ https://issues.apache.org/jira/browse/AIRFLOW-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028218#comment-17028218 ] ASF GitHub Bot commented on AIRFLOW-3349: - sjvanrossum commented on pull request #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in Str… URL: https://github.com/apache/airflow/pull/7329 …eamLogWriter --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > StreamLogWriter encoding is wrong type > -- > > Key: AIRFLOW-3349 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3349 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Affects Versions: 1.10.0 >Reporter: Christopher >Priority: Minor > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > io handlers have encoding that is specified by a string (ie 'UTF-8') or None. > The StreamLogWriter class's encoding attribute is boolean. > > Specifically prevents import of `sympy` from within a PythonOperator, but I > would argue that this implementation should be fixed just so that it can > adhere to the io norm. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in Str…
boring-cyborg[bot] commented on issue #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in Str… URL: https://github.com/apache/airflow/pull/7329#issuecomment-581072311 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sjvanrossum opened a new pull request #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in Str…
sjvanrossum opened a new pull request #7329: [AIRFLOW-3349] Use None instead of False as value for encoding in Str… URL: https://github.com/apache/airflow/pull/7329 …eamLogWriter --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581056087 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=h1) Report > Merging [#7327](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/373c6aa4a208284b5ff72987e4bd8f4e2ada1a1b?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7327 +/- ## = - Coverage 85.87% 85.7% -0.17% = Files 862 863 +1 Lines 40484 40510 +26 = - Hits34767 34721 -46 - Misses 57175789 +72 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `89.11% <100%> (+1.53%)` | :arrow_up: | | [airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==) | `90.16% <88.88%> (+0.13%)` | :arrow_up: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...irflow/example\_dags/example\_kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5) | `15% <0%> (-65.96%)` | :arrow_down: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yX2NvbmZpZy5weQ==) | `16.66% <0%> (-51.76%)` | :arrow_down: | | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: | | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.9% <0%> (-0.44%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=footer). Last update [373c6aa...7511cd8](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6707) Simplify Connection.get_hook method
[ https://issues.apache.org/jira/browse/AIRFLOW-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028213#comment-17028213 ] ASF GitHub Bot commented on AIRFLOW-6707: - mik-laj commented on pull request #7328: [AIRFLOW-6707] Simplify Connection.get_hook method URL: https://github.com/apache/airflow/pull/7328 This method contains a lot of repeated code. I did refactoring and simplified it. By the way, I introduced alphabetical order. In the future, we can expand the plugin mechanism to add support for custom connection types. Now it will be easier. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Simplify Connection.get_hook method > --- > > Key: AIRFLOW-6707 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6707 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7328: [AIRFLOW-6707] Simplify Connection.get_hook method
mik-laj opened a new pull request #7328: [AIRFLOW-6707] Simplify Connection.get_hook method URL: https://github.com/apache/airflow/pull/7328 This method contains a lot of repeated code. I did refactoring and simplified it. By the way, I introduced alphabetical order. In the future, we can expand the plugin mechanism to add support for custom connection types. Now it will be easier. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6707) Simplify Connection.get_hook method
Kamil Bregula created AIRFLOW-6707: -- Summary: Simplify Connection.get_hook method Key: AIRFLOW-6707 URL: https://issues.apache.org/jira/browse/AIRFLOW-6707 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.7 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7319: [AIRFLOW-6702] Dumping kind logs on failure works now
codecov-io commented on issue #7319: [AIRFLOW-6702] Dumping kind logs on failure works now URL: https://github.com/apache/airflow/pull/7319#issuecomment-581069418 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=h1) Report > Merging [#7319](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ba85bde8efaa3e2a87607ddc97fa40a0b7e16e1e?src=pr=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7319 +/- ## == - Coverage 85.86% 85.83% -0.04% == Files 863 863 Lines 4048440521 +37 == + Hits3476334780 +17 - Misses 5721 5741 +20 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.05% <100%> (+0.04%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `90.47% <0%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <0%> (+0.23%)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7319/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=footer). Last update [ba85bde...26f39a4](https://codecov.io/gh/apache/airflow/pull/7319?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#discussion_r373802061 ## File path: airflow/serialization/serialized_objects.py ## @@ -32,8 +32,15 @@ from airflow.serialization.enums import DagAttributeTypes as DAT, Encoding from airflow.serialization.json_schema import Validator, load_dag_schema from airflow.settings import json +from airflow.utils.module_loading import import_string from airflow.www.utils import get_python_source +WHITELIST_OPERATOR_EXTRA_LINKS: List[str] = [ Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#discussion_r373801719 ## File path: airflow/serialization/serialized_objects.py ## @@ -32,8 +32,15 @@ from airflow.serialization.enums import DagAttributeTypes as DAT, Encoding from airflow.serialization.json_schema import Validator, load_dag_schema from airflow.settings import json +from airflow.utils.module_loading import import_string from airflow.www.utils import get_python_source +WHITELIST_OPERATOR_EXTRA_LINKS: List[str] = [ Review comment: I like this name too. It will change in a moment. I was inspired by this https://github.com/apache/airflow/blob/master/airflow/plugins_manager.py#L121 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links
ashb commented on a change in pull request #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#discussion_r373801585 ## File path: airflow/serialization/serialized_objects.py ## @@ -32,8 +32,15 @@ from airflow.serialization.enums import DagAttributeTypes as DAT, Encoding from airflow.serialization.json_schema import Validator, load_dag_schema from airflow.settings import json +from airflow.utils.module_loading import import_string from airflow.www.utils import get_python_source +WHITELIST_OPERATOR_EXTRA_LINKS: List[str] = [ Review comment: ```suggestion BUILTIN_OPERATOR_EXTRA_LINKS: List[str] = [ ``` Its not a quite a whitelist, so how about this as a name? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581067534 > This way we could avoid this lazy-loading for specific imports and have it also solved for all future cases We want to load lazily because some users do not want to use certain operators, so their code should be loaded when it is needed. Now all classes are loaded for each user. When we will transfer this method call, it will still load all classes for all users, not according to the user's needs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581065459 I think this is a completely different problem. I also think it is worth solving it, but it will require more time and testing. If we call the integrate plugins method elsewhere, it will still load GCP and Qubole classes, when it is not needed. In this PR I want to focus only on one small problem and not change the way whole plugins mechanism work. My change is not even related to the plugin loading mechanism at all. This only applies to operators that are in the core. It is a mistake that this code is in the same module. BTW. I think that it will also be much easier to do when we drop plugin support for operators, sensors and hook. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581065459 I think this is a completely different problem. I also think it is worth solving it, but it will require more time and testing. If we call the integrate plugins method elsewhere, it will still load GCP and Qubole classes, when it is not needed. In this PR I want to focus only on one small problem and not change the way whole plugins mechanism work. I think that it will also be much easier to do when we drop plugin support for operators, sensors and hook. This will significantly simplify the code. My change is not even related to the plugin loading mechanism at all. This only applies to operators that are in the core. It is a mistake that this code is in the same module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #6790: [AIRFLOW-6210] Reuse SQLAlchemy session in ti_deps
stale[bot] closed pull request #6790: [AIRFLOW-6210] Reuse SQLAlchemy session in ti_deps URL: https://github.com/apache/airflow/pull/6790 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6210) Reuse SQLAlchemy session in ti_deps
[ https://issues.apache.org/jira/browse/AIRFLOW-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028195#comment-17028195 ] ASF GitHub Bot commented on AIRFLOW-6210: - stale[bot] commented on pull request #6790: [AIRFLOW-6210] Reuse SQLAlchemy session in ti_deps URL: https://github.com/apache/airflow/pull/6790 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Reuse SQLAlchemy session in ti_deps > --- > > Key: AIRFLOW-6210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6210 > Project: Apache Airflow > Issue Type: Bug > Components: core, scheduler >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > This generates unnecessary overhead and it is good practice to use the > session again if possible. These functions are performed in a loop, so it can > be a bigger issues in some cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6698) Allow shorthand notation
[ https://issues.apache.org/jira/browse/AIRFLOW-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-6698. - Resolution: Fixed > Allow shorthand notation > > > Key: AIRFLOW-6698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6698 > Project: Apache Airflow > Issue Type: Sub-task > Components: lineage >Affects Versions: 1.10.7 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Major > Fix For: 1.10.8 > > > Shorthand notation makes it a lot easier to define dags with lineage support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
codecov-io commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581056087 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=h1) Report > Merging [#7327](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/373c6aa4a208284b5ff72987e4bd8f4e2ada1a1b?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7327 +/- ## = - Coverage 85.87% 85.7% -0.17% = Files 862 863 +1 Lines 40484 40510 +26 = - Hits34767 34721 -46 - Misses 57175789 +72 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `89.11% <100%> (+1.53%)` | :arrow_up: | | [airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==) | `90.16% <88.88%> (+0.13%)` | :arrow_up: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...irflow/example\_dags/example\_kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5) | `15% <0%> (-65.96%)` | :arrow_down: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yX2NvbmZpZy5weQ==) | `16.66% <0%> (-51.76%)` | :arrow_down: | | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: | | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.9% <0%> (-0.44%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=footer). Last update [373c6aa...e390d66](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581056087 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=h1) Report > Merging [#7327](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/373c6aa4a208284b5ff72987e4bd8f4e2ada1a1b?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7327 +/- ## = - Coverage 85.87% 85.7% -0.17% = Files 862 863 +1 Lines 40484 40510 +26 = - Hits34767 34721 -46 - Misses 57175789 +72 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `89.11% <100%> (+1.53%)` | :arrow_up: | | [airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==) | `90.16% <88.88%> (+0.13%)` | :arrow_up: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...irflow/example\_dags/example\_kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5) | `15% <0%> (-65.96%)` | :arrow_down: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yX2NvbmZpZy5weQ==) | `16.66% <0%> (-51.76%)` | :arrow_down: | | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: | | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.9% <0%> (-0.44%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=footer). Last update [373c6aa...e390d66](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
codecov-io edited a comment on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581056087 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=h1) Report > Merging [#7327](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/373c6aa4a208284b5ff72987e4bd8f4e2ada1a1b?src=pr=desc) will **decrease** coverage by `0.04%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7327 +/- ## == - Coverage 85.87% 85.83% -0.05% == Files 862 863 +1 Lines 4048440510 +26 == + Hits3476734773 +6 - Misses 5717 5737 +20 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `89.11% <100%> (+1.53%)` | :arrow_up: | | [airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==) | `90.16% <88.88%> (+0.13%)` | :arrow_up: | | [...irflow/example\_dags/example\_kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5) | `15% <0%> (-65.96%)` | :arrow_down: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yX2NvbmZpZy5weQ==) | `16.66% <0%> (-51.76%)` | :arrow_down: | | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96% <0%> (-4%)` | :arrow_down: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `90.47% <0%> (ø)` | :arrow_up: | | [airflow/example\_dags/libs/helper.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvbGlicy9oZWxwZXIucHk=) | `0% <0%> (ø)` | | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7327/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <0%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=footer). Last update [373c6aa...e390d66](https://codecov.io/gh/apache/airflow/pull/7327?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links
potiuk commented on issue #7327: [AIRFLOW-6706] Lazy load operator extra links URL: https://github.com/apache/airflow/pull/7327#issuecomment-581055631 One comment - i think we should not load plugins_manager at all at __init__.py. Why do we do it? Could we remove it from __init__.py altogether and import it when it is really needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-6698) Allow shorthand notation
[ https://issues.apache.org/jira/browse/AIRFLOW-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reassigned AIRFLOW-6698: --- Assignee: Bolke de Bruin > Allow shorthand notation > > > Key: AIRFLOW-6698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6698 > Project: Apache Airflow > Issue Type: Sub-task > Components: lineage >Affects Versions: 1.10.7 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Major > Fix For: 1.10.8 > > > Shorthand notation makes it a lot easier to define dags with lineage support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6706) Lazy load operator extra links
[
https://issues.apache.org/jira/browse/AIRFLOW-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028162#comment-17028162
]
ASF GitHub Bot commented on AIRFLOW-6706:
-
mik-laj commented on pull request #7327: [AIRFLOW-6706] Lazy load operator
extra links
URL: https://github.com/apache/airflow/pull/7327
When we import the airflow package, many modules are loaded, so I looked at
what modules are exactly loaded. I found a lot of classes that should not be
loaded and delay the start of the application very much. I suggest that some
classes be loaded lazily when needed.
Performance benchmark:
```
seq 1 10 | xargs -n 1 -I {} time python -c "import airflow; import sys;
print(len(sys.modules));"
```
Before:
```
1521
3.45 real 1.83 user 0.59 sys
1521
1.86 real 1.60 user 0.40 sys
1521
1.87 real 1.61 user 0.40 sys
1521
1.83 real 1.60 user 0.39 sys
1521
1.84 real 1.62 user 0.40 sys
1521
2.00 real 1.71 user 0.42 sys
1521
1.84 real 1.60 user 0.41 sys
1521
1.84 real 1.60 user 0.40 sys
1521
1.90 real 1.64 user 0.41 sys
1521
1.87 real 1.61 user 0.42 sys
```
After
```
941
1.79 real 0.95 user 0.27 sys
941
1.23 real 0.90 user 0.20 sys
941
1.26 real 0.90 user 0.21 sys
941
1.25 real 0.89 user 0.20 sys
941
1.25 real 0.88 user 0.20 sys
941
1.30 real 0.90 user 0.21 sys
941
1.23 real 0.88 user 0.20 sys
941
1.19 real 0.87 user 0.19 sys
941
1.23 real 0.87 user 0.21 sys
941
1.20 real 0.87 user 0.20 sys
```
Result:
https://user-images.githubusercontent.com/12058428/73596482-a5239a80-4522-11ea-9879-49d158264b81.png;>
and 580 fewer modules - 61%
If anyone is interested, I attach an exact log that shows the import process.
https://gist.github.com/mik-laj/002f5a714c221ba04bc638970094519c
CC: @evgenyshulman
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [X] Description above provides context of the change
- [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [X] Unit tests coverage for changes (not needed for documentation changes)
- [X] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [X] Relevant documentation is updated including usage instructions.
- [X] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lazy load operator extra links
> --
>
> Key: AIRFLOW-6706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6706
> Project: Apache Airflow
> Issue Type: Improvement
> Components: serialization
>Affects Versions: 1.10.7
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7327: [AIRFLOW-6706] Lazy load operator extra links
mik-laj opened a new pull request #7327: [AIRFLOW-6706] Lazy load operator
extra links
URL: https://github.com/apache/airflow/pull/7327
When we import the airflow package, many modules are loaded, so I looked at
what modules are exactly loaded. I found a lot of classes that should not be
loaded and delay the start of the application very much. I suggest that some
classes be loaded lazily when needed.
Performance benchmark:
```
seq 1 10 | xargs -n 1 -I {} time python -c "import airflow; import sys;
print(len(sys.modules));"
```
Before:
```
1521
3.45 real 1.83 user 0.59 sys
1521
1.86 real 1.60 user 0.40 sys
1521
1.87 real 1.61 user 0.40 sys
1521
1.83 real 1.60 user 0.39 sys
1521
1.84 real 1.62 user 0.40 sys
1521
2.00 real 1.71 user 0.42 sys
1521
1.84 real 1.60 user 0.41 sys
1521
1.84 real 1.60 user 0.40 sys
1521
1.90 real 1.64 user 0.41 sys
1521
1.87 real 1.61 user 0.42 sys
```
After
```
941
1.79 real 0.95 user 0.27 sys
941
1.23 real 0.90 user 0.20 sys
941
1.26 real 0.90 user 0.21 sys
941
1.25 real 0.89 user 0.20 sys
941
1.25 real 0.88 user 0.20 sys
941
1.30 real 0.90 user 0.21 sys
941
1.23 real 0.88 user 0.20 sys
941
1.19 real 0.87 user 0.19 sys
941
1.23 real 0.87 user 0.21 sys
941
1.20 real 0.87 user 0.20 sys
```
Result:
https://user-images.githubusercontent.com/12058428/73596482-a5239a80-4522-11ea-9879-49d158264b81.png;>
and 580 fewer modules - 61%
If anyone is interested, I attach an exact log that shows the import process.
https://gist.github.com/mik-laj/002f5a714c221ba04bc638970094519c
CC: @evgenyshulman
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [X] Description above provides context of the change
- [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [X] Unit tests coverage for changes (not needed for documentation changes)
- [X] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [X] Relevant documentation is updated including usage instructions.
- [X] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors
potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors URL: https://github.com/apache/airflow/pull/7323#issuecomment-581050635 All green ! pls take a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7320: [AIRFLOW-6703] The base_log_folder option moved to logging in ci
potiuk commented on issue #7320: [AIRFLOW-6703] The base_log_folder option moved to logging in ci URL: https://github.com/apache/airflow/pull/7320#issuecomment-581050153 Even if not green, this change removes some deprecation logs from our tests which makes it more readable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #7326: [AIRFLOW-6705] One less chatty message
potiuk opened a new pull request #7326: [AIRFLOW-6705] One less chatty message URL: https://github.com/apache/airflow/pull/7326 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6705) Checking for integrations and backends should be less chatty in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028153#comment-17028153 ] ASF GitHub Bot commented on AIRFLOW-6705: - potiuk commented on pull request #7326: [AIRFLOW-6705] One less chatty message URL: https://github.com/apache/airflow/pull/7326 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Checking for integrations and backends should be less chatty in breeze > -- > > Key: AIRFLOW-6705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7326: [AIRFLOW-6705] One less chatty message
potiuk commented on issue #7326: [AIRFLOW-6705] One less chatty message URL: https://github.com/apache/airflow/pull/7326#issuecomment-581050037 @mik-laj- noticed one more extra chatty message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6705) Checking for integrations and backends should be less chatty in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028151#comment-17028151 ] ASF subversion and git services commented on AIRFLOW-6705: -- Commit 39375c01ced008a715bb2a540df31fe2cf908bbb in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=39375c0 ] [AIRFLOW-6705] Less chatty integration/backend checks (#7325) > Checking for integrations and backends should be less chatty in breeze > -- > > Key: AIRFLOW-6705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks in Breeze/CI
potiuk merged pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks in Breeze/CI URL: https://github.com/apache/airflow/pull/7325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6705) Checking for integrations and backends should be less chatty in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028150#comment-17028150 ] ASF GitHub Bot commented on AIRFLOW-6705: - potiuk commented on pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks in Breeze/CI URL: https://github.com/apache/airflow/pull/7325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Checking for integrations and backends should be less chatty in breeze > -- > > Key: AIRFLOW-6705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6698) Allow shorthand notation
[ https://issues.apache.org/jira/browse/AIRFLOW-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028146#comment-17028146 ] ASF subversion and git services commented on AIRFLOW-6698: -- Commit 4c81bcd8601fa08efa570ee231f8f103ef830304 in airflow's branch refs/heads/master from Bolke de Bruin [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4c81bcd ] [AIRFLOW-6698] Add shorthand notation for lineage (#7314) This adds shorthand notation to define dags that have lineage support. | for piping between operators and > and < for setting (static) lineage defintions. > Allow shorthand notation > > > Key: AIRFLOW-6698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6698 > Project: Apache Airflow > Issue Type: Sub-task > Components: lineage >Affects Versions: 1.10.7 >Reporter: Bolke de Bruin >Priority: Major > Fix For: 1.10.8 > > > Shorthand notation makes it a lot easier to define dags with lineage support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6698) Allow shorthand notation
[ https://issues.apache.org/jira/browse/AIRFLOW-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028145#comment-17028145 ] ASF GitHub Bot commented on AIRFLOW-6698: - bolkedebruin commented on pull request #7314: [AIRFLOW-6698] Add shorthand notation for lineage URL: https://github.com/apache/airflow/pull/7314 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow shorthand notation > > > Key: AIRFLOW-6698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6698 > Project: Apache Airflow > Issue Type: Sub-task > Components: lineage >Affects Versions: 1.10.7 >Reporter: Bolke de Bruin >Priority: Major > Fix For: 1.10.8 > > > Shorthand notation makes it a lot easier to define dags with lineage support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] bolkedebruin merged pull request #7314: [AIRFLOW-6698] Add shorthand notation for lineage
bolkedebruin merged pull request #7314: [AIRFLOW-6698] Add shorthand notation for lineage URL: https://github.com/apache/airflow/pull/7314 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6706) Lazy load operator extra links
Kamil Bregula created AIRFLOW-6706: -- Summary: Lazy load operator extra links Key: AIRFLOW-6706 URL: https://issues.apache.org/jira/browse/AIRFLOW-6706 Project: Apache Airflow Issue Type: Improvement Components: serialization Affects Versions: 1.10.7 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6705) Checking for integrations and backends should be less chatty in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028140#comment-17028140 ] ASF GitHub Bot commented on AIRFLOW-6705: - potiuk commented on pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks URL: https://github.com/apache/airflow/pull/7325 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Checking for integrations and backends should be less chatty in breeze > -- > > Key: AIRFLOW-6705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors
potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors URL: https://github.com/apache/airflow/pull/7323#issuecomment-581045923 The sooner we get it - the better - the CRON job overnight will become much more stable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks
potiuk opened a new pull request #7325: [AIRFLOW-6705] Less chatty integration/backend checks URL: https://github.com/apache/airflow/pull/7325 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6705) Checking for integrations and backends should be less chatty in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6705: -- Summary: Checking for integrations and backends should be less chatty in breeze (was: Checking for integrations should be less chatty) > Checking for integrations and backends should be less chatty in breeze > -- > > Key: AIRFLOW-6705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6705) Checking for integrations should be less chatty
Jarek Potiuk created AIRFLOW-6705: - Summary: Checking for integrations should be less chatty Key: AIRFLOW-6705 URL: https://issues.apache.org/jira/browse/AIRFLOW-6705 Project: Apache Airflow Issue Type: Improvement Components: breeze, ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors
potiuk commented on issue #7323: [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors URL: https://github.com/apache/airflow/pull/7323#issuecomment-581039319 We had quite a lot of failures in cron master job (where the Dockerfile is rebuilt) and the reason was RAT download failures - it was from maven repos and those repos were not stable, so I figured we need to do it "properly" using abache backup servers + apache mirrors. It should succeed (as it does not touch .py nor kubernetes related files, so we can merge it even before we find and fix the kubernetes/kerberos failures. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[
https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028115#comment-17028115
]
ASF GitHub Bot commented on AIRFLOW-6704:
-
yuqian90 commented on pull request #7324: [AIRFLOW-6704] Set
TaskInstance.operator in constructor
URL: https://github.com/apache/airflow/pull/7324
``TaskInstance.operator`` is currently only set when task is executed. But
if a task is marked success or failed, the ``operator`` field is left as None.
This causes bugs when some other code tries to use the ``operator`` field to
find the name of the class.
The fix is trivial, just set ``TaskInstance.operator`` in its constructor.
An assertion is also added in ``test_mark_tasks.py`` to test this.
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> TaskInstance.operator is not set when task is marked success or failed in the
> Web UI
>
>
> Key: AIRFLOW-6704
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6704
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
>
> {{TaskInstance.operator}} is currently only set when task is executed. But if
> a task is marked success or failed, the {{operator}} field is left as
> {{None}}.
> This causes bugs when some code tries to use the operator field to find the
> name of the class.
> The fix is trivial, just set {{TaskInstance.operator}} in its constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] yuqian90 opened a new pull request #7324: [AIRFLOW-6704] Set TaskInstance.operator in constructor
yuqian90 opened a new pull request #7324: [AIRFLOW-6704] Set TaskInstance.operator in constructor URL: https://github.com/apache/airflow/pull/7324 ``TaskInstance.operator`` is currently only set when task is executed. But if a task is marked success or failed, the ``operator`` field is left as None. This causes bugs when some other code tries to use the ``operator`` field to find the name of the class. The fix is trivial, just set ``TaskInstance.operator`` in its constructor. An assertion is also added in ``test_mark_tasks.py`` to test this. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7314: [AIRFLOW-6698] Add shorthand notation for lineage
potiuk commented on a change in pull request #7314: [AIRFLOW-6698] Add
shorthand notation for lineage
URL: https://github.com/apache/airflow/pull/7314#discussion_r373784742
##
File path: airflow/providers/papermill/operators/papermill.py
##
@@ -45,19 +45,26 @@ class PapermillOperator(BaseOperator):
:param parameters: the notebook parameters to set
:type parameters: dict
"""
+supports_lineage = True
+
@apply_defaults
def __init__(self,
- input_nb: str,
- output_nb: str,
- parameters: Dict,
+ input_nb: Optional[str] = None,
+ output_nb: Optional[str] = None,
+ parameters: Optional[Dict] = None,
*args, **kwargs) -> None:
super().__init__(*args, **kwargs)
-self.inlets.append(NoteBook(url=input_nb,
-parameters=parameters))
-self.outlets.append(NoteBook(url=output_nb))
+if input_nb:
+self.inlets.append(NoteBook(url=input_nb,
+parameters=parameters))
+if output_nb:
+self.outlets.append(NoteBook(url=output_nb))
def execute(self, context):
+if not self.inlets or not self.outlets:
+raise ValueError("Input notebook or output notebook is not
specified")
Review comment:
I think it's ok to have it at execute time - no matter if we use builder
pattern or the shorthand operators, it cannot be done init().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] XD-DENG opened a new issue #238: Typo in community page
XD-DENG opened a new issue #238: Typo in community page URL: https://github.com/apache/airflow-site/issues/238 Typo in page https://airflow.apache.org/community/ https://user-images.githubusercontent.com/11539188/73594121-6fbf8280-450b-11ea-8243-9bc11c0fa7ac.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bolkedebruin commented on a change in pull request #7314: [AIRFLOW-6698] Add shorthand notation for lineage
bolkedebruin commented on a change in pull request #7314: [AIRFLOW-6698] Add
shorthand notation for lineage
URL: https://github.com/apache/airflow/pull/7314#discussion_r373783904
##
File path: airflow/providers/papermill/operators/papermill.py
##
@@ -45,19 +45,26 @@ class PapermillOperator(BaseOperator):
:param parameters: the notebook parameters to set
:type parameters: dict
"""
+supports_lineage = True
+
@apply_defaults
def __init__(self,
- input_nb: str,
- output_nb: str,
- parameters: Dict,
+ input_nb: Optional[str] = None,
+ output_nb: Optional[str] = None,
+ parameters: Optional[Dict] = None,
*args, **kwargs) -> None:
super().__init__(*args, **kwargs)
-self.inlets.append(NoteBook(url=input_nb,
-parameters=parameters))
-self.outlets.append(NoteBook(url=output_nb))
+if input_nb:
+self.inlets.append(NoteBook(url=input_nb,
+parameters=parameters))
+if output_nb:
+self.outlets.append(NoteBook(url=output_nb))
def execute(self, context):
+if not self.inlets or not self.outlets:
+raise ValueError("Input notebook or output notebook is not
specified")
Review comment:
I agree but can you propose how to do that if with short hand notation
inlets/outlets are set after initialization has taken place of the operator?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
Qian Yu created AIRFLOW-6704:
Summary: TaskInstance.operator is not set when task is marked
success or failed in the Web UI
Key: AIRFLOW-6704
URL: https://issues.apache.org/jira/browse/AIRFLOW-6704
Project: Apache Airflow
Issue Type: Improvement
Components: models
Affects Versions: 1.10.7
Reporter: Qian Yu
{{TaskInstance.operator}} is currently only set when task is executed. But if a
task is marked success or failed, the {{operator}} field is left as {{None}}.
This causes bugs when some code tries to use the operator field to find the
name of the class.
The fix is trivial, just set {{TaskInstance.operator}} in its constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[
https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qian Yu reassigned AIRFLOW-6704:
Assignee: Qian Yu
> TaskInstance.operator is not set when task is marked success or failed in the
> Web UI
>
>
> Key: AIRFLOW-6704
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6704
> Project: Apache Airflow
> Issue Type: Improvement
> Components: models
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
>
> {{TaskInstance.operator}} is currently only set when task is executed. But if
> a task is marked success or failed, the {{operator}} field is left as
> {{None}}.
> This causes bugs when some code tries to use the operator field to find the
> name of the class.
> The fix is trivial, just set {{TaskInstance.operator}} in its constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[
https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qian Yu updated AIRFLOW-6704:
-
Issue Type: Bug (was: Improvement)
> TaskInstance.operator is not set when task is marked success or failed in the
> Web UI
>
>
> Key: AIRFLOW-6704
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6704
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
>
> {{TaskInstance.operator}} is currently only set when task is executed. But if
> a task is marked success or failed, the {{operator}} field is left as
> {{None}}.
> This causes bugs when some code tries to use the operator field to find the
> name of the class.
> The fix is trivial, just set {{TaskInstance.operator}} in its constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7323: [AURFLOW-6701] Rat is downloaded from stable backup/mirrors
potiuk opened a new pull request #7323: [AURFLOW-6701] Rat is downloaded from stable backup/mirrors URL: https://github.com/apache/airflow/pull/7323 Also curl options are now using long format and include --fail to protect against some temporary errors (5xx). Also RAT download uses now two possible sources of downloads and fallbacks to the second if first is not available. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6593) Google Cloud Stackdriver Alerting Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028108#comment-17028108 ] ASF GitHub Bot commented on AIRFLOW-6593: - jithin97 commented on pull request #7322: [AIRFLOW-6593] [WIP] Add GCP Stackdriver Alerting Hooks and Operators URL: https://github.com/apache/airflow/pull/7322 ### Description Add GCP Stackdriver Alerting Hooks and Operators --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Google Cloud Stackdriver Alerting Operators > --- > > Key: AIRFLOW-6593 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6593 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Affects Versions: 1.10.7 >Reporter: Jithin Sukumar >Assignee: Jithin Sukumar >Priority: Minor > > Add stackdriver alerting operators which can perform the following functions: > # Regarding policies > ## Add new alerting policy/policies > ## Delete existing policy/policies > ## Update existing policy/policies > # Regarding notification channels > ## Add new alerting channel > ## Delete existing channel > ## Update existing channel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] jithin97 opened a new pull request #7322: [AIRFLOW-6593] [WIP] Add GCP Stackdriver Alerting Hooks and Operators
jithin97 opened a new pull request #7322: [AIRFLOW-6593] [WIP] Add GCP Stackdriver Alerting Hooks and Operators URL: https://github.com/apache/airflow/pull/7322 ### Description Add GCP Stackdriver Alerting Hooks and Operators --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7321: [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring
ashb commented on a change in pull request #7321: [AIRFLOW-] Remove duplication in BaseOperator docstring URL: https://github.com/apache/airflow/pull/7321#discussion_r373782834 ## File path: airflow/models/baseoperator.py ## @@ -173,8 +173,6 @@ class derived from this one results in the creation of a task object, DAGS. Options can be set as string or using the constants defined in the static class ``airflow.utils.WeightRule`` :type weight_rule: str -:param queue: specifies which task queue to use Review comment: Good spot! But could you please remove the other one? This one matches the position of the queue param This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7321: [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring
potiuk commented on issue #7321: [AIRFLOW-] Remove duplication in BaseOperator docstring URL: https://github.com/apache/airflow/pull/7321#issuecomment-581032700 Thanks @zuku1985 ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zuku1985 opened a new pull request #7321: [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring
zuku1985 opened a new pull request #7321: [AIRFLOW-] Remove duplication in BaseOperator docstring URL: https://github.com/apache/airflow/pull/7321 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Minor change in BaseOperator docs. I removed duplicated queue parameter. Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7309: [AIRFLOW-4681] Make sensors module pylint compatible
potiuk commented on issue #7309: [AIRFLOW-4681] Make sensors module pylint compatible URL: https://github.com/apache/airflow/pull/7309#issuecomment-581032385 Yep. I am working on the build errors in master still. I will let you know when to rebase when it is all fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6175: [AIRFLOW-5545] Fixes recursion in DAG cycle testing
potiuk commented on issue #6175: [AIRFLOW-5545] Fixes recursion in DAG cycle testing URL: https://github.com/apache/airflow/pull/6175#issuecomment-581032191 I think it's nice to fix that one @dimberman ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
ashb commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r373780705 ## File path: tests/providers/amazon/aws/hooks/test_athena.py ## @@ -0,0 +1,120 @@ +# -*- coding: utf-8 -*- Review comment: Do we really not have any existing tests for the Athena hook? (i.e. Why is this a new file?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
