[jira] [Updated] (AIRFLOW-6566) Replace uses of imp still left with importlib.

[Kousuke Saruta (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kousuke Saruta updated AIRFLOW-6566:

Summary: Replace uses of imp still left with importlib.  (was: Replace uses 
of imp still left with importlib)

> Replace uses of imp still left with importlib.
> --
>
> Key: AIRFLOW-6566
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6566
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: 1.10.8
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
>
> I've replaced a use of `imp` with `importlib` in AIRFLOW-6507 but I noticed 
> are still two uses left.
> Let's replace them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6566) Replace uses of imp still left with importlib

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015704#comment-17015704
 ] 

ASF GitHub Bot commented on AIRFLOW-6566:
-

sarutak commented on pull request #7174: [AIRFLOW-6566] Replace uses of imp 
still left with importlib
URL: https://github.com/apache/airflow/pull/7174
 
 
   I've replaced a use of `imp` with `importlib` in #7099 but I noticed are 
still two uses left.
   Let's replace them.
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Replace uses of imp still left with importlib
> -
>
> Key: AIRFLOW-6566
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6566
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: 1.10.8
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
>
> I've replaced a use of `imp` with `importlib` in AIRFLOW-6507 but I noticed 
> are still two uses left.
> Let's replace them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] sarutak opened a new pull request #7174: [AIRFLOW-6566] Replace uses of imp still left with importlib

[GitBox]

sarutak opened a new pull request #7174: [AIRFLOW-6566] Replace uses of imp 
still left with importlib
URL: https://github.com/apache/airflow/pull/7174
 
 
   I've replaced a use of `imp` with `importlib` in #7099 but I noticed are 
still two uses left.
   Let's replace them.
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6566) Replace uses of imp still left with importlib

[Kousuke Saruta (Jira)]

Kousuke Saruta created AIRFLOW-6566:
---

 Summary: Replace uses of imp still left with importlib
 Key: AIRFLOW-6566
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6566
 Project: Apache Airflow
  Issue Type: Improvement
  Components: plugins
Affects Versions: 1.10.8
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta


I've replaced a use of `imp` with `importlib` but I noticed are still two uses 
left.
Let's replace them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6566) Replace uses of imp still left with importlib

[Kousuke Saruta (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kousuke Saruta updated AIRFLOW-6566:

Description: 
I've replaced a use of `imp` with `importlib` in AIRFLOW-6507 but I noticed are 
still two uses left.
Let's replace them.

  was:
I've replaced a use of `imp` with `importlib` but I noticed are still two uses 
left.
Let's replace them.


> Replace uses of imp still left with importlib
> -
>
> Key: AIRFLOW-6566
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6566
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: 1.10.8
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
>
> I've replaced a use of `imp` with `importlib` in AIRFLOW-6507 but I noticed 
> are still two uses left.
> Let's replace them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] sarutak commented on issue #7099: [AIRFLOW-6507] Replace the use of imp.load_source with another solution.

[GitBox]

sarutak commented on issue #7099: [AIRFLOW-6507] Replace the use of 
imp.load_source with another solution.
URL: https://github.com/apache/airflow/pull/7099#issuecomment-574537650
 
 
   I noticed there are 2 `imp` left...
   I'll open another PR to fix them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox]

codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs 
when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#issuecomment-573300577
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@689f025`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `87.75%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7128/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7128   +/-   ##
   =
 Coverage  ?   85.13%   
   =
 Files ?  710   
 Lines ?39515   
 Branches  ?0   
   =
 Hits  ?33641   
 Misses? 5874   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5)
 | `75% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9nY3NfdG9fZ2NzLnB5)
 | `100% <ø> (ø)` | |
   | 
[...le\_dags/example\_passing\_params\_via\_test\_command.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9wYXNzaW5nX3BhcmFtc192aWFfdGVzdF9jb21tYW5kLnB5)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_latest\_only.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seS5weQ==)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.81% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_gcs\_to\_sftp.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9nY3NfdG9fc2Z0cC5weQ==)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=)
 | `63.33% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_trigger\_target\_dag.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX3RhcmdldF9kYWcucHk=)
 | `90% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | |
   | ... and [141 
more](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=footer). 
Last update 
[689f025...5f12283](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox]

codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs 
when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#issuecomment-573300577
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@689f025`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `87.75%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7128/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7128   +/-   ##
   =
 Coverage  ?   85.13%   
   =
 Files ?  710   
 Lines ?39515   
 Branches  ?0   
   =
 Hits  ?33641   
 Misses? 5874   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5)
 | `75% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9nY3NfdG9fZ2NzLnB5)
 | `100% <ø> (ø)` | |
   | 
[...le\_dags/example\_passing\_params\_via\_test\_command.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9wYXNzaW5nX3BhcmFtc192aWFfdGVzdF9jb21tYW5kLnB5)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_latest\_only.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seS5weQ==)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.81% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_gcs\_to\_sftp.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9nY3NfdG9fc2Z0cC5weQ==)
 | `100% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=)
 | `63.33% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_trigger\_target\_dag.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX3RhcmdldF9kYWcucHk=)
 | `90% <ø> (ø)` | |
   | 
[airflow/example\_dags/example\_branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | |
   | ... and [141 
more](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=footer). 
Last update 
[689f025...5f12283](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 removed a comment on issue #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

tooptoop4 removed a comment on issue #7157: [AIRFLOW-6251] add config for max 
tasks per dag
URL: https://github.com/apache/airflow/pull/7157#issuecomment-574418492
 
 
   @zhongjiajie any idea on why test not working?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6565) BigQuery - replace deprecated connection parameters

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015689#comment-17015689
 ] 

ASF GitHub Bot commented on AIRFLOW-6565:
-

TobKed commented on pull request #7173: [AIRFLOW-6565] BigQuery - replace 
deprecated connection parameters
URL: https://github.com/apache/airflow/pull/7173
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BigQuery - replace deprecated connection parameters
> ---
>
> Key: AIRFLOW-6565
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6565
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, operators
>Affects Versions: 1.10.7
>Reporter: Tobiasz Kedzierski
>Assignee: Tobiasz Kedzierski
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] TobKed opened a new pull request #7173: [AIRFLOW-6565] BigQuery - replace deprecated connection parameters

[GitBox]

TobKed opened a new pull request #7173: [AIRFLOW-6565] BigQuery - replace 
deprecated connection parameters
URL: https://github.com/apache/airflow/pull/7173
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6560) db password leaks to logs

[Jira]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcin Jasiński updated AIRFLOW-6560:
-
Description: 
I have configured Airflow metadata db as MySQL.
{code:java}
sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
After that I have used initdb command:
{code:java}
airflow initdb{code}
Tables in the airflow db have been created.

Then I have runed commands:
{code:java}
airflow scheduler
airflow webserver -p 8080{code}
in logs:
{code:java}
[2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
password as a plain text.

Probably comes that from Celery - 
[https://github.com/celery/celery/blob/master/celery/worker/consumer/connection.py#L24]

  was:
I have configured Airflow metadata db as MySQL.
{code:java}
sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
After that I have used initdb command:
{code:java}
airflow initdb{code}
Tables in the airflow db have been created.

Then I have runed commands:
{code:java}
airflow scheduler
airflow webserver -p 8080{code}
in logs:
{code:java}
[2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
password as a plain text.


> db password leaks to logs
> -
>
> Key: AIRFLOW-6560
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6560
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging, security
>Affects Versions: 1.10.6
>Reporter: Marcin Jasiński
>Priority: Critical
>
> I have configured Airflow metadata db as MySQL.
> {code:java}
> sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
> After that I have used initdb command:
> {code:java}
> airflow initdb{code}
> Tables in the airflow db have been created.
> Then I have runed commands:
> {code:java}
> airflow scheduler
> airflow webserver -p 8080{code}
> in logs:
> {code:java}
> [2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
> sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
> password as a plain text.
> Probably comes that from Celery - 
> [https://github.com/celery/celery/blob/master/celery/worker/consumer/connection.py#L24]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6565) BigQuery - replace deprecated parameters with correct ones

[Tobiasz Kedzierski (Jira)]

Tobiasz Kedzierski created AIRFLOW-6565:
---

 Summary: BigQuery - replace deprecated parameters with correct ones
 Key: AIRFLOW-6565
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6565
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp, operators
Affects Versions: 1.10.7
Reporter: Tobiasz Kedzierski
Assignee: Tobiasz Kedzierski






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6565) BigQuery - replace deprecated connection parameters

[Tobiasz Kedzierski (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tobiasz Kedzierski updated AIRFLOW-6565:

Summary: BigQuery - replace deprecated connection parameters  (was: 
BigQuery - replace deprecated parameters with correct ones)

> BigQuery - replace deprecated connection parameters
> ---
>
> Key: AIRFLOW-6565
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6565
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, operators
>Affects Versions: 1.10.7
>Reporter: Tobiasz Kedzierski
>Assignee: Tobiasz Kedzierski
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6560) db password leaks to logs

[Jira]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcin Jasiński updated AIRFLOW-6560:
-
Description: 
I have configured Airflow metadata db as MySQL.
{code:java}
sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
After that I have used initdb command:
{code:java}
airflow initdb{code}
Tables in the airflow db have been created.

Then I have runed commands:
{code:java}
airflow scheduler
airflow webserver -p 8080{code}
in logs:
{code:java}
[2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
password as a plain text.

  was:
I have configured Airflow metadata db as MySQL.
{code:java}
sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
After that I have used initdb command:
{code:java}
airflow initdb{code}
Tables in the airflow db have been created.

Then I have runed command:
{code:java}
airflow webserver -p 8080{code}
in logs:
{code:java}
[2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
password as a plain text.


> db password leaks to logs
> -
>
> Key: AIRFLOW-6560
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6560
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging, security
>Affects Versions: 1.10.6
>Reporter: Marcin Jasiński
>Priority: Critical
>
> I have configured Airflow metadata db as MySQL.
> {code:java}
> sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
> After that I have used initdb command:
> {code:java}
> airflow initdb{code}
> Tables in the airflow db have been created.
> Then I have runed commands:
> {code:java}
> airflow scheduler
> airflow webserver -p 8080{code}
> in logs:
> {code:java}
> [2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
> sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
> password as a plain text.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox]

sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle 
error occurs when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#discussion_r366722592
 
 

 ##
 File path: .travis.yml
 ##
 @@ -87,6 +91,18 @@ jobs:
 BACKEND=sqlite
 PYTHON_VERSION=3.6
   stage: test
+- name: "Tests [Sqlite][3.7][multiprocessing(spawn)]"
+  env: >-
+AIRFLOW__CORE__MP_START_METHOD=spawn
 
 Review comment:
   @potiuk I tried to do what you suggested but I noticed It's not enough 
because the pickle error happens when scheduler runs.
   Instead, I added one test case to `tests/jobs/test_scheduler_job.py`, which 
is almost a copy of an existing test case but set `mp_start_method` to `spawn`.
   The pickle error can be tested the test case but another issue, updated 
configuration is not inherited by child processes, cannot be tested so I also 
added a test case to `tests/utils/test_dag_processing.py`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366714188
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -938,17 +838,93 @@ The 'properties' and 'jars' properties for the Dataproc 
related operators (`Data
 and `dataproc_jars`respectively.
 Arguments for dataproc_properties dataproc_jars
 
-### Failure callback will be called when task is marked failed
-When task is marked failed by user or task fails due to system failures - on 
failure call back will be called as part of clean up
+## Airflow 1.10.7
 
-See [AIRFLOW-5621](https://jira.apache.org/jira/browse/AIRFLOW-5621) for 
details
+### Changes in experimental API execution_date microseconds replacement
+
+The default behavior was to strip the microseconds (and milliseconds, etc) off 
of all dag runs triggered by
+by the experimental REST API.  The default behavior will change when an 
explicit execution_date is
+passed in the request body.  It will also now be possible to have the 
execution_date generated, but
+keep the microseconds by sending `replace_microseconds=false` in the request 
body.  The default
+behavior can be overridden by sending `replace_microseconds=true` along with 
an explicit execution_date
+
+### Viewer won't have edit permissions on DAG view.
+
+### Renamed "extra" requirements for cloud providers
+
+Subpackages for specific services have been combined into one variant for
+each cloud provider. The name of the subpackage for the Google Cloud Platform
+has changed to follow style.
+
+If you want to install integration for Microsoft Azure, then instead of
+```
+pip install 
'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'
+```
+you should execute `pip install 'apache-airflow[azure]'`
+
+If you want to install integration for Amazon Web Services, then instead of
+`pip install 'apache-airflow[s3,emr]'`, you should execute `pip install 
'apache-airflow[aws]'`
+
+If you want to install integration for Google Cloud Platform, then instead of
+`pip install 'apache-airflow[gcp_api]'`, you should execute `pip install 
'apache-airflow[gcp]'`.
+The old way will work until the release of Airflow 2.1.
+
+## Airflow 1.10.6
+
+### BaseOperator::render_template function signature changed
+
+Previous versions of the `BaseOperator::render_template` function required an 
`attr` argument as the first
+positional argument, along with `content` and `context`. This function 
signature was changed in 1.10.6 and
+the `attr` argument is no longer required (or accepted).
+
+In order to use this function in subclasses of the `BaseOperator`, the `attr` 
argument must be removed:
+```python
+result = self.render_template('myattr', self.myattr, context)  # Pre-1.10.6 
call
+...
+result = self.render_template(self.myattr, context)  # Post-1.10.6 call
+```
+
+### Changes to `aws_default` Connection's default region
+
+The region of Airflow's default connection to AWS (`aws_default`) was 
previously
+set to `us-east-1` during installation.
+
+The region now needs to be set manually, either in the connection screens in
+Airflow, via the `~/.aws` config files, or via the `AWS_DEFAULT_REGION` 
environment
+variable.
+
+### Some DAG Processing metrics have been renamed
+
+The following metrics are deprecated and won't be emitted in Airflow 2.0:
+
+- `scheduler.dagbag.errors` and `dagbag_import_errors` -- use 
`dag_processing.import_errors` instead
+- `dag_file_processor_timeouts` -- use `dag_processing.processor_timeouts` 
instead
+- `collect_dags` -- use `dag_processing.total_parse_time` instead
+- `dag.loading-duration.` -- use 
`dag_processing.last_duration.` instead
+- `dag_processing.last_runtime.` -- use 
`dag_processing.last_duration.` instead
 
 Review comment:
   Correct


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366703132
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -938,17 +838,93 @@ The 'properties' and 'jars' properties for the Dataproc 
related operators (`Data
 and `dataproc_jars`respectively.
 Arguments for dataproc_properties dataproc_jars
 
-### Failure callback will be called when task is marked failed
-When task is marked failed by user or task fails due to system failures - on 
failure call back will be called as part of clean up
+## Airflow 1.10.7
 
-See [AIRFLOW-5621](https://jira.apache.org/jira/browse/AIRFLOW-5621) for 
details
+### Changes in experimental API execution_date microseconds replacement
+
+The default behavior was to strip the microseconds (and milliseconds, etc) off 
of all dag runs triggered by
+by the experimental REST API.  The default behavior will change when an 
explicit execution_date is
+passed in the request body.  It will also now be possible to have the 
execution_date generated, but
+keep the microseconds by sending `replace_microseconds=false` in the request 
body.  The default
+behavior can be overridden by sending `replace_microseconds=true` along with 
an explicit execution_date
+
+### Viewer won't have edit permissions on DAG view.
+
+### Renamed "extra" requirements for cloud providers
+
+Subpackages for specific services have been combined into one variant for
+each cloud provider. The name of the subpackage for the Google Cloud Platform
+has changed to follow style.
+
+If you want to install integration for Microsoft Azure, then instead of
+```
+pip install 
'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'
+```
+you should execute `pip install 'apache-airflow[azure]'`
+
+If you want to install integration for Amazon Web Services, then instead of
+`pip install 'apache-airflow[s3,emr]'`, you should execute `pip install 
'apache-airflow[aws]'`
+
+If you want to install integration for Google Cloud Platform, then instead of
+`pip install 'apache-airflow[gcp_api]'`, you should execute `pip install 
'apache-airflow[gcp]'`.
+The old way will work until the release of Airflow 2.1.
+
+## Airflow 1.10.6
+
+### BaseOperator::render_template function signature changed
+
+Previous versions of the `BaseOperator::render_template` function required an 
`attr` argument as the first
+positional argument, along with `content` and `context`. This function 
signature was changed in 1.10.6 and
+the `attr` argument is no longer required (or accepted).
+
+In order to use this function in subclasses of the `BaseOperator`, the `attr` 
argument must be removed:
+```python
+result = self.render_template('myattr', self.myattr, context)  # Pre-1.10.6 
call
+...
+result = self.render_template(self.myattr, context)  # Post-1.10.6 call
+```
+
+### Changes to `aws_default` Connection's default region
+
+The region of Airflow's default connection to AWS (`aws_default`) was 
previously
+set to `us-east-1` during installation.
+
+The region now needs to be set manually, either in the connection screens in
+Airflow, via the `~/.aws` config files, or via the `AWS_DEFAULT_REGION` 
environment
+variable.
+
+### Some DAG Processing metrics have been renamed
+
+The following metrics are deprecated and won't be emitted in Airflow 2.0:
+
+- `scheduler.dagbag.errors` and `dagbag_import_errors` -- use 
`dag_processing.import_errors` instead
+- `dag_file_processor_timeouts` -- use `dag_processing.processor_timeouts` 
instead
+- `collect_dags` -- use `dag_processing.total_parse_time` instead
+- `dag.loading-duration.` -- use 
`dag_processing.last_duration.` instead
+- `dag_processing.last_runtime.` -- use 
`dag_processing.last_duration.` instead
 
 ## Airflow 1.10.5
 
 No breaking changes.
 
 ## Airflow 1.10.4
 
+### Export MySQL timestamps as UTC
+
+`MySqlToGoogleCloudStorageOperator` now exports TIMESTAMP columns as UTC
+by default, rather than using the default timezone of the MySQL server.
+This is the correct behavior for use with BigQuery, since BigQuery
+assumes that TIMESTAMP columns without time zones are in UTC. To
+preserve the previous behavior, set `ensure_utc` to `False.`
+
+### Changes to GoogleCloudStorageHook
+
+* The following parameters have been replaced in all the methods in GCSHook:
+  * `bucket` is changed to `bucket_name`
+  * `object` is changed to `object_name`
+
+* The `maxResults` parameter in `GoogleCloudStorageHook.list` has been renamed 
to `max_results` for consistency.
+
 
 Review comment:
   This is not true. We only change the parameter names on Master, check: 
https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/hooks/gcs_hook.py#L298-L343


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the 

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366702859
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -938,17 +838,93 @@ The 'properties' and 'jars' properties for the Dataproc 
related operators (`Data
 and `dataproc_jars`respectively.
 Arguments for dataproc_properties dataproc_jars
 
-### Failure callback will be called when task is marked failed
-When task is marked failed by user or task fails due to system failures - on 
failure call back will be called as part of clean up
+## Airflow 1.10.7
 
-See [AIRFLOW-5621](https://jira.apache.org/jira/browse/AIRFLOW-5621) for 
details
+### Changes in experimental API execution_date microseconds replacement
+
+The default behavior was to strip the microseconds (and milliseconds, etc) off 
of all dag runs triggered by
+by the experimental REST API.  The default behavior will change when an 
explicit execution_date is
+passed in the request body.  It will also now be possible to have the 
execution_date generated, but
+keep the microseconds by sending `replace_microseconds=false` in the request 
body.  The default
+behavior can be overridden by sending `replace_microseconds=true` along with 
an explicit execution_date
+
+### Viewer won't have edit permissions on DAG view.
+
+### Renamed "extra" requirements for cloud providers
+
+Subpackages for specific services have been combined into one variant for
+each cloud provider. The name of the subpackage for the Google Cloud Platform
+has changed to follow style.
+
+If you want to install integration for Microsoft Azure, then instead of
+```
+pip install 
'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'
+```
+you should execute `pip install 'apache-airflow[azure]'`
+
+If you want to install integration for Amazon Web Services, then instead of
+`pip install 'apache-airflow[s3,emr]'`, you should execute `pip install 
'apache-airflow[aws]'`
+
+If you want to install integration for Google Cloud Platform, then instead of
+`pip install 'apache-airflow[gcp_api]'`, you should execute `pip install 
'apache-airflow[gcp]'`.
+The old way will work until the release of Airflow 2.1.
+
+## Airflow 1.10.6
+
+### BaseOperator::render_template function signature changed
+
+Previous versions of the `BaseOperator::render_template` function required an 
`attr` argument as the first
+positional argument, along with `content` and `context`. This function 
signature was changed in 1.10.6 and
+the `attr` argument is no longer required (or accepted).
+
+In order to use this function in subclasses of the `BaseOperator`, the `attr` 
argument must be removed:
+```python
+result = self.render_template('myattr', self.myattr, context)  # Pre-1.10.6 
call
+...
+result = self.render_template(self.myattr, context)  # Post-1.10.6 call
+```
+
+### Changes to `aws_default` Connection's default region
+
+The region of Airflow's default connection to AWS (`aws_default`) was 
previously
+set to `us-east-1` during installation.
+
+The region now needs to be set manually, either in the connection screens in
+Airflow, via the `~/.aws` config files, or via the `AWS_DEFAULT_REGION` 
environment
+variable.
+
+### Some DAG Processing metrics have been renamed
+
+The following metrics are deprecated and won't be emitted in Airflow 2.0:
+
+- `scheduler.dagbag.errors` and `dagbag_import_errors` -- use 
`dag_processing.import_errors` instead
+- `dag_file_processor_timeouts` -- use `dag_processing.processor_timeouts` 
instead
+- `collect_dags` -- use `dag_processing.total_parse_time` instead
+- `dag.loading-duration.` -- use 
`dag_processing.last_duration.` instead
+- `dag_processing.last_runtime.` -- use 
`dag_processing.last_duration.` instead
 
 ## Airflow 1.10.5
 
 No breaking changes.
 
 ## Airflow 1.10.4
 
+### Export MySQL timestamps as UTC
+
+`MySqlToGoogleCloudStorageOperator` now exports TIMESTAMP columns as UTC
+by default, rather than using the default timezone of the MySQL server.
+This is the correct behavior for use with BigQuery, since BigQuery
+assumes that TIMESTAMP columns without time zones are in UTC. To
+preserve the previous behavior, set `ensure_utc` to `False.`
 
 Review comment:
   This is correct


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366702370
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1043,6 +1019,12 @@ dag.get_task_instances(session=your_session)
 
 ## Airflow 1.10.3
 
+### New `dag_discovery_safe_mode` config option
+
+If `dag_discovery_safe_mode` is enabled, only check files for DAGs if
+they contain the strings "airflow" and "DAG". For backwards
+compatibility, this option is enabled by default.
+
 
 Review comment:
   This is already in `Airflow 1.10.3` section. Check 
https://github.com/apache/airflow/pull/7166/files#diff-bb7ee25332d5fdca3ead2f545d72b359R1061-R1065


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366701958
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1226,6 +1208,11 @@ generates has been fixed.
 
 ## Airflow 1.10.2
 
+### New `dag_processor_manager_log_location` config option
+
+The DAG parsing manager log now by default will be log into a file, where its 
location is
+controlled by the new `dag_processor_manager_log_location` config option in 
core section.
+
 
 Review comment:
   Correct


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366701810
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1305,6 +1292,11 @@ or enabled autodetect of schema:
 
 ## Airflow 1.10.1
 
+### min_file_parsing_loop_time config option temporarily disabled
+
+The scheduler.min_file_parsing_loop_time config option has been temporarily 
removed due to
+some bugs.
+
 
 Review comment:
   This is correct, good catch


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366655381
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
 
 
 Review comment:
   from 5528:
When the end_of_log_mark is wrapped in a log record, the 
end_of_log_mark can no longer be 
able to match the log line in _read:
   
   metadata['end_of_log'] = False if not logs \
   else logs[-1].message == self.end_of_log_mark.strip()
It leads to the UI keeps calling backend and generates lots of load to 
ES.
   By removing the log_id from the end-of-log mark, it would make it worse as 
the ui would continue to try to find the end-of-log mark and it won't ever find 
it as it searches the end-of-log mark by log_id.
   
   I am not sure what  the sentence mean by "When the end_of_log_mark is 
wrapped in a log record". I also observed that the end-of-log mark might end up 
within the same line of other log lines and it would prevent us from finding 
the end-of-log mark in those cases. To address that, I always add an obnoxious 
print right in front of the end-of-log mark line, to ensure the "end-of-log" 
mark is always in a separate line when printing to console. This is import for 
filebeat/logstash on kubernetes to pick up the end-of-log mark log line in a 
separate document. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366699596
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1336,6 +1328,20 @@ Installation and upgrading requires setting 
`SLUGIFY_USES_TEXT_UNIDECODE=yes` in
 `AIRFLOW_GPL_UNIDECODE=yes`. In case of the latter a GPL runtime dependency 
will be installed due to a
 dependency (python-nvd3 -> python-slugify -> unidecode).
 
+### Removed deprecated import mechanism
+
+The deprecated import mechanism has been removed so the import of modules 
becomes more consistent and explicit.
+
+For example: `from airflow.operators import BashOperator`
+becomes `from airflow.operators.bash_operator import BashOperator`
+
+### Changes to sensor imports
+
+Sensors are now accessible via `airflow.sensors` and no longer via 
`airflow.operators.sensors`.
+
+For example: `from airflow.operators.sensors import BaseSensorOperator`
+becomes `from airflow.sensors.base_sensor_operator import BaseSensorOperator`
+
 
 Review comment:
   For reference, check: 
https://github.com/apache/airflow/blob/1.10.7/airflow/operators/__init__.py


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7166: [AIRFLOW-XXXX] Move UPDATING changes into correct versions

[GitBox]

kaxil commented on a change in pull request #7166: [AIRFLOW-] Move UPDATING 
changes into correct versions
URL: https://github.com/apache/airflow/pull/7166#discussion_r366699525
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1336,6 +1328,20 @@ Installation and upgrading requires setting 
`SLUGIFY_USES_TEXT_UNIDECODE=yes` in
 `AIRFLOW_GPL_UNIDECODE=yes`. In case of the latter a GPL runtime dependency 
will be installed due to a
 dependency (python-nvd3 -> python-slugify -> unidecode).
 
+### Removed deprecated import mechanism
+
+The deprecated import mechanism has been removed so the import of modules 
becomes more consistent and explicit.
+
+For example: `from airflow.operators import BashOperator`
+becomes `from airflow.operators.bash_operator import BashOperator`
+
+### Changes to sensor imports
+
+Sensors are now accessible via `airflow.sensors` and no longer via 
`airflow.operators.sensors`.
+
+For example: `from airflow.operators.sensors import BaseSensorOperator`
+becomes `from airflow.sensors.base_sensor_operator import BaseSensorOperator`
+
 
 Review comment:
   This is still working on 1.10.7 too, so it was rightly placed in Master.
   
   This deprecation would be for 2.0


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to 
end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-573571156
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=h1) 
Report
   > Merging 
[#7141](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d654d69d7794a57c5c51685a8a96f1d7c38c2302?src=pr=desc)
 will **decrease** coverage by `0.12%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7141/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7141  +/-   ##
   ==
   - Coverage   85.24%   85.12%   -0.13% 
   ==
 Files 683  710  +27 
 Lines   3915539483 +328 
   ==
   + Hits3337833609 +231 
   - Misses   5777 5874  +97
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `70.21% <0%> (-1.53%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `92.66% <80%> (-0.74%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.31% <0%> (-20.49%)` | :arrow_down: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `91.59% <0%> (-0.49%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | ... and [81 
more](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=footer). 
Last update 
[d654d69...75be1e1](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to 
end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-573571156
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=h1) 
Report
   > Merging 
[#7141](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d654d69d7794a57c5c51685a8a96f1d7c38c2302?src=pr=desc)
 will **decrease** coverage by `0.12%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7141/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7141  +/-   ##
   ==
   - Coverage   85.24%   85.12%   -0.13% 
   ==
 Files 683  710  +27 
 Lines   3915539483 +328 
   ==
   + Hits3337833609 +231 
   - Misses   5777 5874  +97
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `70.21% <0%> (-1.53%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `92.66% <80%> (-0.74%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.31% <0%> (-20.49%)` | :arrow_down: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `91.59% <0%> (-0.49%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | ... and [81 
more](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=footer). 
Last update 
[d654d69...75be1e1](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add 
config for max tasks per dag
URL: https://github.com/apache/airflow/pull/7157#discussion_r31407
 
 

 ##
 File path: tests/models/test_dagbag.py
 ##
 @@ -151,6 +151,42 @@ def test_zip(self):
 dagbag.process_file(os.path.join(TEST_DAGS_FOLDER, "test_zip.zip"))
 self.assertTrue(dagbag.get_dag("test_zip_dag"))
 
+@conf_vars({('core', 'max_tasks_per_dag'): '5'})
+def test_process_file_max_task_check(self):
+"""
+test if num_tasks > max_tasks_per_dag can be identified
+"""
+a_dag_id = "example_short_circuit_operator"
 
 Review comment:
   ```suggestion
   a_dag_id = "test_example_bash_operator"
   ```
   
   I take a quick look. `TEST_DAGS_FOLDER` point to 
`airflow/tests/models/../dags` and this path not including dag 
`example_short_circuit_operator`. So you could not process_file 
`airflow/tests/models/../dags/example_short_circuit_operator.py`
   
   Maybe you should use others dag in `airflow/tests/models/../dags`(which 
`TEST_DAGS_FOLDER` point to), just like `test_example_bash_operator` and change 
`@conf_vars({('core', 'max_tasks_per_dag'): '5'})` to `@conf_vars({('core', 
'max_tasks_per_dag'): '7'})`.
   
   So as others test @tooptoop4 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6564) Display extra diagnostics if initial environment check fails

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015560#comment-17015560
 ] 

ASF GitHub Bot commented on AIRFLOW-6564:
-

potiuk commented on pull request #7172: [AIRFLOW-6564] Additional diagnostics 
information on CI check failure
URL: https://github.com/apache/airflow/pull/7172
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Display extra diagnostics if initial environment check fails
> 
>
> Key: AIRFLOW-6564
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6564
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk opened a new pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure

[GitBox]

potiuk opened a new pull request #7172: [AIRFLOW-6564] Additional diagnostics 
information on CI check failure
URL: https://github.com/apache/airflow/pull/7172
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6564) Display extra diagnostics if initial environment check fails

[Jarek Potiuk (Jira)]

Jarek Potiuk created AIRFLOW-6564:
-

 Summary: Display extra diagnostics if initial environment check 
fails
 Key: AIRFLOW-6564
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6564
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 2.0.0
Reporter: Jarek Potiuk






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #7171: [AIRFLOW-6563] Add end_date for marked tasks

[GitBox]

codecov-io edited a comment on issue #7171: [AIRFLOW-6563] Add end_date for 
marked tasks
URL: https://github.com/apache/airflow/pull/7171#issuecomment-574426943
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=h1) 
Report
   > Merging 
[#7171](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/f7ee687cf9b796685dee0d8ba3142b570248b6e1?src=pr=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7171/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master#7171 +/-   ##
   =
   - Coverage85.4%   85.11%   -0.3% 
   =
 Files 710  710 
 Lines   3947939482  +3 
   =
   - Hits3371833606-112 
   - Misses   5761 5876+115
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `95.48% <100%> (+0.08%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.31% <0%> (-20.49%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `90.72% <0%> (-1.16%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=footer). 
Last update 
[f7ee687...d0ce89d](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7171: [AIRFLOW-6563] Add end_date for marked tasks

[GitBox]

codecov-io edited a comment on issue #7171: [AIRFLOW-6563] Add end_date for 
marked tasks
URL: https://github.com/apache/airflow/pull/7171#issuecomment-574426943
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=h1) 
Report
   > Merging 
[#7171](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/f7ee687cf9b796685dee0d8ba3142b570248b6e1?src=pr=desc)
 will **decrease** coverage by `0.74%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7171/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7171  +/-   ##
   ==
   - Coverage85.4%   84.66%   -0.75% 
   ==
 Files 710  710  
 Lines   3947939482   +3 
   ==
   - Hits3371833427 -291 
   - Misses   5761 6055 +294
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `95.48% <100%> (+0.08%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5)
 | `52.94% <0%> (-32.36%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=footer). 
Last update 
[f7ee687...d0ce89d](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add 
config for max tasks per dag
URL: https://github.com/apache/airflow/pull/7157#discussion_r31407
 
 

 ##
 File path: tests/models/test_dagbag.py
 ##
 @@ -151,6 +151,42 @@ def test_zip(self):
 dagbag.process_file(os.path.join(TEST_DAGS_FOLDER, "test_zip.zip"))
 self.assertTrue(dagbag.get_dag("test_zip_dag"))
 
+@conf_vars({('core', 'max_tasks_per_dag'): '5'})
+def test_process_file_max_task_check(self):
+"""
+test if num_tasks > max_tasks_per_dag can be identified
+"""
+a_dag_id = "example_short_circuit_operator"
 
 Review comment:
   ```suggestion
   a_dag_id = "test_example_bash_operator"
   ```
   
   I take a quick look. `TEST_DAGS_FOLDER` point to 
`airflow/tests/models/../dags` and this path not including dag 
`example_short_circuit_operator`. So you could not process_file 
`airflow/tests/models/../dags/example_short_circuit_operator.py`
   
   Maybe you should use others dag in `airflow/tests/models/../dags`(which 
`TEST_DAGS_FOLDER` point to), just like `test_example_bash_operator` and change 
`@conf_vars({('core', 'max_tasks_per_dag'): '5'})` to `@conf_vars({('core', 
'max_tasks_per_dag'): '7'})`.
   
   So as others test


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add 
config for max tasks per dag
URL: https://github.com/apache/airflow/pull/7157#discussion_r30224
 
 

 ##
 File path: airflow/models/dagbag.py
 ##
 @@ -300,26 +302,37 @@ def process_file(self, filepath, only_if_updated=True, 
safe_mode=True):
 dag.full_filepath = filepath
 if dag.fileloc != filepath and not is_zipfile:
 dag.fileloc = filepath
-try:
-dag.is_subdag = False
-self.bag_dag(dag, parent_dag=dag, root_dag=dag)
-if isinstance(dag._schedule_interval, str):
-croniter(dag._schedule_interval)
-found_dags.append(dag)
-found_dags += dag.subdags
-except (CroniterBadCronError,
-CroniterBadDateError,
-CroniterNotAlphaError) as cron_e:
-self.log.exception("Failed to bag_dag: %s", 
dag.full_filepath)
+
+num_tasks = len(dag.tasks)
+if max_tasks_per_dag > 0 and num_tasks > max_tasks_per_dag:
 
 Review comment:
   But code base already have some example and IDE like pycharm will give a 
hint change to `var1 < var2 < var3`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

zhongjiajie commented on a change in pull request #7157: [AIRFLOW-6251] add 
config for max tasks per dag
URL: https://github.com/apache/airflow/pull/7157#discussion_r366360841
 
 

 ##
 File path: airflow/models/dagbag.py
 ##
 @@ -300,26 +302,37 @@ def process_file(self, filepath, only_if_updated=True, 
safe_mode=True):
 dag.full_filepath = filepath
 if dag.fileloc != filepath and not is_zipfile:
 dag.fileloc = filepath
-try:
-dag.is_subdag = False
-self.bag_dag(dag, parent_dag=dag, root_dag=dag)
-if isinstance(dag._schedule_interval, str):
-croniter(dag._schedule_interval)
-found_dags.append(dag)
-found_dags += dag.subdags
-except (CroniterBadCronError,
-CroniterBadDateError,
-CroniterNotAlphaError) as cron_e:
-self.log.exception("Failed to bag_dag: %s", 
dag.full_filepath)
+
+num_tasks = len(dag.tasks)
+if max_tasks_per_dag > 0 and num_tasks > max_tasks_per_dag:
 
 Review comment:
   Really? but I remember we have a PR to do that, I will take a look


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file 
mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-574451555
 
 
   > Also regarding adding index my **5c**:
   > Currently we have something like:
   > `GET /search?q=user:kimchy` (To search all indices in a cluster, omit the 
parameter.)
   > New logic:
   > `GET /kimchy,elasticsearch/_search?q=user:kimchy` (search in 1 or several 
incedes), which means all old code won't work.
   > 
   > I like idea to add index explicitly in search param, but it should be 
optional than required.
   
   I changed the current test cases to cover this case by using the index 
"test_index" in the test cases through the new parameter to __init()


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366655381
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
 
 
 Review comment:
   from 5528:
When the end_of_log_mark is wrapped in a log record, the 
end_of_log_mark can no longer be 
able to match the log line in _read:
   
   metadata['end_of_log'] = False if not logs \
   else logs[-1].message == self.end_of_log_mark.strip()
It leads to the UI keeps calling backend and generates lots of load to 
ES.
   By removing the log_id from the end-of-log mark, it would make it worse as 
the ui would continue to try to find the end-of-log mark and it won't ever find 
it as it searches the end-of-log mark by log_id.
   
   I am not sure what  the sentence mean by "When the end_of_log_mark is 
wrapped in a log record". I also observed that the end-of-log mark might end up 
within the same line of other log lines and it would prevent us from find the 
end-of-log mark in those cases. To address that, I always add an obnoxious 
print right in front of the end-of-log mark line, to ensure the "end-of-log" 
mark is always in a separate line when printing to console. This is import for 
filebeat/logstash on kubernetes to pick up the end-of-log mark log line in a 
separate document. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file 
mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-574448853
 
 
   > @larryzhu2018 Yup, this is two tickets as Andrii mentions.
   > 
   > And this seems to revert the change from #6159 -- so you will need to 
explain this change in much more detail as to why you think it your version is 
now right.
   
   This change reverts #6159 and I validated that after reverting that, the 
end-to-end scenarios work and airflow-web can find the end-of-log mark using 
the log-id.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366652482
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
 
 
 Review comment:
   sorry I meant that I sent the email to Andrii and the two authors of 5528. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file 
mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-574447922
 
 
   > @ashb IMO better to split 2 unrelated changes into 2 tickets:
   > 
   > * add index (good idea)
   > * fix bug another ticket?
   >   general note: i would like to see test case to proof bug and improve 
test coverage after this change.
   
   The fix for "5528" removed the log_id from the end-of-log mark log record. 
It broke the end  to end scenarios. Because the current elastic search mock 
does not add log-id to the log records, so it cannot test the end-to-end 
scenarios where in the real elastic search cluster we rely on the log_id to 
find the logs for airflow.  Since you were the developer who added log_id 
support, perhaps you can help to update the elastic search mock to add the log 
id in the log records, so that the scenarios won't be broken in the future, 
e.g. by another fix along the line of 5528?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7170: [AIRFLOW-6562] mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[GitBox]

codecov-io edited a comment on issue #7170: [AIRFLOW-6562] mushroom cloud error 
when clicking 'mark failed/success' from graph view of dag that has never been 
run yet
URL: https://github.com/apache/airflow/pull/7170#issuecomment-574447220
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@00c6758`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `95.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7170/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7170   +/-   ##
   =
 Coverage  ?   85.12%   
   =
 Files ?  710   
 Lines ?39482   
 Branches  ?0   
   =
 Hits  ?33608   
 Misses? 5874   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.25% <ø> (ø)` | |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <ø> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.81% <ø> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `94.92% <ø> (ø)` | |
   | 
[...flow/contrib/example\_dags/example\_gcs\_to\_gdrive.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djc190b19nZHJpdmUucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2JxLnB5)
 | `70.58% <100%> (ø)` | |
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `94.83% <100%> (ø)` | |
   | 
[...irflow/contrib/operators/gcp\_container\_operator.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfY29udGFpbmVyX29wZXJhdG9yLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/hooks/gcp\_container\_hook.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9jb250YWluZXJfaG9vay5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | |
   | ... and [18 
more](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=footer). 
Last update 
[00c6758...2eff0e7](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7170: [AIRFLOW-6562] mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[GitBox]

codecov-io commented on issue #7170: [AIRFLOW-6562] mushroom cloud error when 
clicking 'mark failed/success' from graph view of dag that has never been run 
yet
URL: https://github.com/apache/airflow/pull/7170#issuecomment-574447220
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@00c6758`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `95.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7170/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7170   +/-   ##
   =
 Coverage  ?   84.66%   
   =
 Files ?  710   
 Lines ?39482   
 Branches  ?0   
   =
 Hits  ?33426   
 Misses? 6056   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.25% <ø> (ø)` | |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <ø> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.81% <ø> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `94.92% <ø> (ø)` | |
   | 
[...flow/contrib/example\_dags/example\_gcs\_to\_gdrive.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djc190b19nZHJpdmUucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2JxLnB5)
 | `70.58% <100%> (ø)` | |
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `94.83% <100%> (ø)` | |
   | 
[...irflow/contrib/operators/gcp\_container\_operator.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfY29udGFpbmVyX29wZXJhdG9yLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/hooks/gcp\_container\_hook.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9jb250YWluZXJfaG9vay5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | |
   | ... and [18 
more](https://codecov.io/gh/apache/airflow/pull/7170/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=footer). 
Last update 
[00c6758...2eff0e7](https://codecov.io/gh/apache/airflow/pull/7170?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on issue #7141: [AIRFLOW-6544] add log_id to end-of-file 
mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#issuecomment-574446137
 
 
   > Also regarding adding index my **5c**:
   > Currently we have something like:
   > `GET /search?q=user:kimchy` (To search all indices in a cluster, omit the 
parameter.)
   > New logic:
   > `GET /kimchy,elasticsearch/_search?q=user:kimchy` (search in 1 or several 
incedes), which means all old code won't work.
   > 
   > I like idea to add index explicitly in search param, but it should be 
optional than required.
   
   I just set the default index "*" so that by default you do not need to set 
an index name.
I am not sure I understand the queries above but to search in a specific 
index, you just need to run /_search?q=user:kimchy, instead of 
/_search?q=user:kimchy


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366648976
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
 
 Review comment:
   This is to make sure the "end-of-log" mark log record will always be in a 
separate line therefore a separate log record. as those like us work with 
elastic search for logging, for various reasons, it is not always easy to 
ensure log lines are not combined. Because the end-of-log mark is so special, 
it helps with the print() here to guarantee that we can always find the mark 
with the log-id from es_task_handler.py.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366645903
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
 
 
 Review comment:
   yes, I sent email to them and suggested those changes are bad and copied 
that message to you. Please see email I sent to you and authors of 5528.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs

[GitBox]

larryzhu2018 commented on a change in pull request #7141: [AIRFLOW-6544] add 
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r366647600
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -255,7 +256,9 @@ def close(self):
 
 # Mark the end of file using end of log mark,
 # so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
 
 Review comment:
   it is import for us to find the log_id mark in the log line with 
"end_of_log_mark" from the elastic search cluster. What I observed was that I 
saw the end of log mark can end up with the same line of the previous log lines 
hence it would prevent us from finding the end-of-log mark in some cases 
console prints from random places without the newline. I am adding an obnoxious 
new line (print()) so as to guarantee that end-of-log mark is a separate log 
record. For any other log line it is actually benign to have two log lines to 
combine into one line in elastic search. Only the end-of-log mark absolutely 
need to be in its own line.  This is just to make the solution here more robust 
and decoupled from the rest of log lines. I understand this is a fix for 
reliability and it probably is not very clean.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

[GitBox]

codecov-io commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when 
clicking 'task instance details' from graph view of dag that has never been run 
yet
URL: https://github.com/apache/airflow/pull/7169#issuecomment-574431885
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@00c6758`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `96.02%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7169/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7169   +/-   ##
   =
 Coverage  ?   85.12%   
   =
 Files ?  710   
 Lines ?39479   
 Branches  ?0   
   =
 Hits  ?33606   
 Misses? 5873   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.25% <ø> (ø)` | |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <ø> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.81% <ø> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `94.92% <ø> (ø)` | |
   | 
[...flow/contrib/example\_dags/example\_gcs\_to\_gdrive.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djc190b19nZHJpdmUucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2JxLnB5)
 | `70.58% <100%> (ø)` | |
   | 
[...irflow/contrib/operators/gcp\_container\_operator.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfY29udGFpbmVyX29wZXJhdG9yLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/hooks/gcp\_container\_hook.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9jb250YWluZXJfaG9vay5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/sensors/file\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZmlsZV9zZW5zb3IucHk=)
 | `100% <100%> (ø)` | |
   | ... and [18 
more](https://codecov.io/gh/apache/airflow/pull/7169/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=footer). 
Last update 
[00c6758...0e97674](https://codecov.io/gh/apache/airflow/pull/7169?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on a change in pull request #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

yuqian90 commented on a change in pull request #7119: [AIRFLOW-5840] Add 
operator extra link to external task sensor
URL: https://github.com/apache/airflow/pull/7119#discussion_r366633960
 
 

 ##
 File path: airflow/sensors/external_task_sensor.py
 ##
 @@ -16,22 +16,71 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
 import datetime
 import os
-from typing import Optional, Union
+from typing import FrozenSet, Optional, Union
 
 from sqlalchemy import func
 
+from airflow.configuration import conf
 from airflow.exceptions import AirflowException
-from airflow.models import DagBag, DagModel, DagRun, TaskInstance
+from airflow.models import BaseOperatorLink, DagBag, DagModel, DagRun, 
TaskInstance
 from airflow.operators.dummy_operator import DummyOperator
 from airflow.sensors.base_sensor_operator import BaseSensorOperator
 from airflow.utils.decorators import apply_defaults
 from airflow.utils.session import provide_session
 from airflow.utils.state import State
 
 
+def get_possible_target_execution_dates(execution_date, execution_delta, 
execution_date_fn):
+"""
+Gets the execution date(s) of an external DAG for which an
+ExternalTaskSensor should succeed on. Default is the execution
+date itself, but it may be modified if a non-null execution delta
+or execution date function is passed in.
+
+:param execution_date: The execution date of the sensor
+:type execution_date: datetime.datetime
+:param execution_delta: Time difference between the sensor
+execution date and the target DAG run execution date. Positive
+delta looks back in time.
+:type execution_delta: Optional[datetime.timedelta]
+:param execution_date_fn: Function to compute the execution date(s)
+of the target DAG run to look at given the sensor's execution
+date.
+:type execution_date_fn: Optional[Callable]
+:return: Execution date(s) to wait for
+:rtype: List[datetime.datetime]
+"""
+if execution_delta:
+dttm = execution_date - execution_delta
+elif execution_date_fn:
+dttm = execution_date_fn(execution_date)
+else:
+dttm = execution_date
+
+return dttm if isinstance(dttm, list) else [dttm]
+
+
+class ExternalTaskLink(BaseOperatorLink):
+name = 'External DAG'
+
+def get_link(self, operator, dttm):
+possible_execution_dates = get_possible_target_execution_dates(
+execution_date=dttm,
+execution_delta=getattr(operator, 'execution_delta', None),
+execution_date_fn=None,
 
 Review comment:
   Oh that's unfortunate if we can't make an existing `execution_date_fn` 
feature work here because I assume many people would need to be using 
`execution_date_fn` to shift execution_date around according to some fancy 
logic they may have.
   
   I have one suggestion that might work. If we know the `task_id` and `dag_id` 
of the `operator`, we can always find the real `ExternalTaskSensor` object 
(e.g. by looking it up in the `DagBag`). Once we have a reference to the real 
`ExternalTaskSensor` object, we can call its `execution_date_fn`.
   
   If that doesn't work for you, I think it's important to make sure that if 
the `ExternalTaskSensor` uses `execution_date_fn`, the extra link doesn't get 
generated. In the current state of the PR, it looks like the link is going to 
point to the wrong `execution_date` if the user intended to use 
`execution_date_fn`. That is going to cause confusion. Also a comment that says 
`execution_date_fn` is not supported for the extra link is probably helpful too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox]

sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle 
error occurs when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#discussion_r366632753
 
 

 ##
 File path: .travis.yml
 ##
 @@ -87,6 +91,18 @@ jobs:
 BACKEND=sqlite
 PYTHON_VERSION=3.6
   stage: test
+- name: "Tests [Sqlite][3.7][multiprocessing(spawn)]"
+  env: >-
+AIRFLOW__CORE__MP_START_METHOD=spawn
 
 Review comment:
   Ah, I understand. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6233) Task get stuck on scheduled on LocalExecutor

[Robert Conroy (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015492#comment-17015492
 ] 

Robert Conroy commented on AIRFLOW-6233:


[~BrechtDeVlieger] I've been unable to reproduce this locally. Do you see this 
on the first DAG run, or should I be waiting for a future run?

> Task get stuck on scheduled on LocalExecutor
> 
>
> Key: AIRFLOW-6233
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6233
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Brecht De Vlieger
>Priority: Blocker
> Attachments: DAG.png, airflow.cfg, expected_gantt.png, logs.tar.gz, 
> observed_gantt.png, test.py
>
>
> Use case description:
> One DAG with a sequence of tasks run by the LocalExecutor. The number of 
> parallel tasks (6) to be run is higher than the concurrency limit (3) 
> (parallelism and dag_concurrency). An image of the DAG layout is attached 
> below.
> Expected behaviour:
> Scheduler uses the three available slots to run as many tasks concurrently as 
> possible. The DAG finishes in some finite time without restarting the 
> scheduler. Example Gantt chart attached below.
> Observed behaviour:
> The tasks are not optimally scheduled and after a while the tasks no longer 
> move from the 'scheduled' state to 'queued'.  When the scheduler is restarted 
> the tasks are picked up again as expected. The Gantt chart is also in the 
> attachments. Note that the chart does not show the tasks that never get 
> scheduled. You can find the logs in the attachments.
> How to reproduce:
> Install vanilla airflow version 1.10.6 and run with LocalExecutor (I used a 
> mysql database). The airflow.cfg and dag file are added to the attachments. 
> The problem does not exist in 1.10.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #7171: [AIRFLOW-6563] Add end_date for marked tasks

[GitBox]

codecov-io commented on issue #7171: [AIRFLOW-6563] Add end_date for marked 
tasks
URL: https://github.com/apache/airflow/pull/7171#issuecomment-574426943
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=h1) 
Report
   > Merging 
[#7171](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/f7ee687cf9b796685dee0d8ba3142b570248b6e1?src=pr=desc)
 will **decrease** coverage by `52.58%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7171/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#7171   +/-   ##
   ===
   - Coverage85.4%   32.82%   -52.59% 
   ===
 Files 710  709-1 
 Lines   3947939469   -10 
   ===
   - Hits3371812955-20763 
   - Misses   576126514+20753
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `18.7% <0%> (-76.69%)` | :arrow_down: |
   | 
[...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcGlnX29wZXJhdG9yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/operators/file\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9maWxlX3RvX2djcy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/gcp\_spanner\_hook.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9zcGFubmVyX2hvb2sucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV9vcGVyYXRvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [577 
more](https://codecov.io/gh/apache/airflow/pull/7171/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=footer). 
Last update 
[f7ee687...46c3b8b](https://codecov.io/gh/apache/airflow/pull/7171?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on issue #7157: [AIRFLOW-6251] add config for max tasks per dag

[GitBox]

tooptoop4 commented on issue #7157: [AIRFLOW-6251] add config for max tasks per 
dag
URL: https://github.com/apache/airflow/pull/7157#issuecomment-574418492
 
 
   @zhongjiajie any idea on why test not working?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6563) User marked tasks does not have a valid end_date

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015470#comment-17015470
 ] 

ASF GitHub Bot commented on AIRFLOW-6563:
-

YingboWang commented on pull request #7171: [AIRFLOW-6563] Add end_date for 
marked tasks
URL: https://github.com/apache/airflow/pull/7171
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   Airflow UI mark_tasks does not update the end_date for task_instance when 
marking them to a final state. This PR add the end_date and duration setting 
for user marked ti state. 
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> User marked tasks does not have a valid end_date
> 
>
> Key: AIRFLOW-6563
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6563
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.10.7
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Minor
> Fix For: master
>
>
> When user mark task instance to success/failed from UI, the marked task did 
> not update a valid end_date. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] YingboWang opened a new pull request #7171: [AIRFLOW-6563] Add end_date for marked tasks

[GitBox]

YingboWang opened a new pull request #7171: [AIRFLOW-6563] Add end_date for 
marked tasks
URL: https://github.com/apache/airflow/pull/7171
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   Airflow UI mark_tasks does not update the end_date for task_instance when 
marking them to a final state. This PR add the end_date and duration setting 
for user marked ti state. 
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-6563) User marked tasks does not have a valid end_date

[Yingbo Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-6563 started by Yingbo Wang.

> User marked tasks does not have a valid end_date
> 
>
> Key: AIRFLOW-6563
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6563
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.10.7
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Minor
> Fix For: master
>
>
> When user mark task instance to success/failed from UI, the marked task did 
> not update a valid end_date. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tooptoop4 opened a new pull request #7170: [AIRFLOW-6562] mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[GitBox]

tooptoop4 opened a new pull request #7170: [AIRFLOW-6562] mushroom cloud error 
when clicking 'mark failed/success' from graph view of dag that has never been 
run yet
URL: https://github.com/apache/airflow/pull/7170
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015460#comment-17015460
 ] 

ASF GitHub Bot commented on AIRFLOW-6562:
-

tooptoop4 commented on pull request #7170: [AIRFLOW-6562] mushroom cloud error 
when clicking 'mark failed/success' from graph view of dag that has never been 
run yet
URL: https://github.com/apache/airflow/pull/7170
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> mushroom cloud error when clicking 'mark failed/success' from graph view of 
> dag that has never been run yet
> ---
>
> Key: AIRFLOW-6562
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6562
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.6
> Environment: localexec, mysql metastore, 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> # create a new dag
>  # go to graph view
>  # click on one of the tasks (it should have a white border)
>  # click on 'past/future' on either 2nd last row (mark failed) or last row 
> (mark success)
>  # then click either (mark failed) or (mark success)
> below error appears
> Traceback (most recent call last):
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in 
> wsgi_app
>  response = self.full_dispatch_request()
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in 
> full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in 
> handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in 
> reraise
>  raise value
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in 
> full_dispatch_request
>  rv = self.dispatch_request()
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in 
> dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, 
> in inner
>  return self._run_view(f, *args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, 
> in _run_view
>  return fn(self, *args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 
> 258, in decorated_view
>  return func(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 
> 290, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 
> 337, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 
> 1449, in failed
>  future, past, State.FAILED)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 
> 1420, in _mark_task_instance_state
>  commit=False)
>  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, 
> in wrapper
>  return 

[jira] [Work started] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[t oo (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-6562 started by t oo.
-
> mushroom cloud error when clicking 'mark failed/success' from graph view of 
> dag that has never been run yet
> ---
>
> Key: AIRFLOW-6562
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6562
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.6
> Environment: localexec, mysql metastore, 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> # create a new dag
>  # go to graph view
>  # click on one of the tasks (it should have a white border)
>  # click on 'past/future' on either 2nd last row (mark failed) or last row 
> (mark success)
>  # then click either (mark failed) or (mark success)
> below error appears
> Traceback (most recent call last):
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in 
> wsgi_app
>  response = self.full_dispatch_request()
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in 
> full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in 
> handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in 
> reraise
>  raise value
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in 
> full_dispatch_request
>  rv = self.dispatch_request()
>  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in 
> dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, 
> in inner
>  return self._run_view(f, *args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, 
> in _run_view
>  return fn(self, *args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 
> 258, in decorated_view
>  return func(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 
> 290, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 
> 337, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 
> 1449, in failed
>  future, past, State.FAILED)
>  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 
> 1420, in _mark_task_instance_state
>  commit=False)
>  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, 
> in wrapper
>  return func(*args, **kwargs)
>  File 
> "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
>  line 105, in set_state
>  dates = get_execution_dates(dag, execution_date, future, past)
>  File 
> "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
>  line 246, in get_execution_dates
>  raise ValueError("Received non-localized date {}".format(execution_date))
> ValueError: Received non-localized date 2020-01-14T21:58:44.855743+00:00
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6563) User marked tasks does not have a valid end_date

[Yingbo Wang (Jira)]

Yingbo Wang created AIRFLOW-6563:


 Summary: User marked tasks does not have a valid end_date
 Key: AIRFLOW-6563
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6563
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 1.10.7
Reporter: Yingbo Wang
Assignee: Yingbo Wang
 Fix For: master


When user mark task instance to success/failed from UI, the marked task did not 
update a valid end_date. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[t oo (Jira)]

t oo created AIRFLOW-6562:
-

 Summary: mushroom cloud error when clicking 'mark failed/success' 
from graph view of dag that has never been run yet
 Key: AIRFLOW-6562
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6562
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui
Affects Versions: 1.10.6
 Environment: localexec, mysql metastore, 1.10.6
Reporter: t oo
Assignee: t oo


# create a new dag
 # go to graph view
 # click on one of the tasks (it should have a white border)
 # click on 'task instance details'

below error appears

 

Traceback (most recent call last):
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 2446, in wsgi_app
 response = self.full_dispatch_request()
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1951, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1820, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1949, in full_dispatch_request
 rv = self.dispatch_request()
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1935, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py", 
line 69, in inner
 return self._run_view(f, *args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py", 
line 368, in _run_view
 return fn(self, *args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py", 
line 261, in decorated_view
 return func(*args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py", 
line 295, in wrapper
 return f(*args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py", 
line 995, in task
 dep_context=dep_context)]
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
 line 643, in get_failed_dep_statuses
 dep_context):
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py",
 line 106, in get_dep_statuses
 for dep_status in self._get_dep_statuses(ti, session, dep_context):
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py",
 line 50, in _get_dep_statuses
 if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no attribute 'run_id'

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[t oo (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

t oo updated AIRFLOW-6562:
--
Description: 
# create a new dag
 # go to graph view
 # click on one of the tasks (it should have a white border)
 # click on 'past/future' on either 2nd last row (mark failed) or last row 
(mark success)
 # then click either (mark failed) or (mark success)

below error appears

Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in 
wsgi_app
 response = self.full_dispatch_request()
 File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in 
full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in 
handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in 
reraise
 raise value
 File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in 
full_dispatch_request
 rv = self.dispatch_request()
 File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in 
dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in 
inner
 return self._run_view(f, *args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, 
in _run_view
 return fn(self, *args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, 
in decorated_view
 return func(*args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 290, 
in wrapper
 return f(*args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 337, 
in wrapper
 return f(*args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 1449, 
in failed
 future, past, State.FAILED)
 File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 1420, 
in _mark_task_instance_state
 commit=False)
 File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in 
wrapper
 return func(*args, **kwargs)
 File 
"/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
 line 105, in set_state
 dates = get_execution_dates(dag, execution_date, future, past)
 File 
"/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
 line 246, in get_execution_dates
 raise ValueError("Received non-localized date {}".format(execution_date))
ValueError: Received non-localized date 2020-01-14T21:58:44.855743+00:00

 

 

  was:
# create a new dag
 # go to graph view
 # click on one of the tasks (it should have a white border)
 # click on 'task instance details'

below error appears

 

Traceback (most recent call last):
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 2446, in wsgi_app
 response = self.full_dispatch_request()
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1951, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1820, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1949, in full_dispatch_request
 rv = self.dispatch_request()
 File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
line 1935, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py", 
line 69, in inner
 return self._run_view(f, *args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py", 
line 368, in _run_view
 return fn(self, *args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py", 
line 261, in decorated_view
 return func(*args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py", 
line 295, in wrapper
 return f(*args, **kwargs)
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py", 
line 995, in task
 dep_context=dep_context)]
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
 line 643, in get_failed_dep_statuses
 dep_context):
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py",
 line 106, in get_dep_statuses
 for dep_status in self._get_dep_statuses(ti, session, dep_context):
 File 
"/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py",
 line 50, in _get_dep_statuses
 if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no 

[jira] [Commented] (AIRFLOW-6555) mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015430#comment-17015430
 ] 

ASF GitHub Bot commented on AIRFLOW-6555:
-

tooptoop4 commented on pull request #7169: [AIRFLOW-6555] mushroom cloud error 
when clicking 'task instance details' from graph view of dag that has never 
been run yet
URL: https://github.com/apache/airflow/pull/7169
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> mushroom cloud error when clicking 'task instance details' from graph view of 
> dag that has never been run yet
> -
>
> Key: AIRFLOW-6555
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6555
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.6
> Environment: localexec, mysql metastore, 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> # create a new dag
>  # go to graph view
>  # click on one of the tasks (it should have a white border)
>  # click on 'task instance details'
> below error appears
>  
> Traceback (most recent call last):
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 2446, in wsgi_app
>  response = self.full_dispatch_request()
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1951, in full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1820, in handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1949, in full_dispatch_request
>  rv = self.dispatch_request()
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1935, in dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
>  line 69, in inner
>  return self._run_view(f, *args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
>  line 368, in _run_view
>  return fn(self, *args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py",
>  line 261, in decorated_view
>  return func(*args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py", 
> line 295, in wrapper
>  return f(*args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py", 
> line 995, in task
>  dep_context=dep_context)]
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
>  line 643, in get_failed_dep_statuses
>  dep_context):
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py",
>  line 106, in get_dep_statuses
>  for dep_status in self._get_dep_statuses(ti, session, dep_context):
>  File 
> 

[GitHub] [airflow] tooptoop4 opened a new pull request #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

[GitBox]

tooptoop4 opened a new pull request #7169: [AIRFLOW-6555] mushroom cloud error 
when clicking 'task instance details' from graph view of dag that has never 
been run yet
URL: https://github.com/apache/airflow/pull/7169
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-6555) mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

[t oo (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-6555 started by t oo.
-
> mushroom cloud error when clicking 'task instance details' from graph view of 
> dag that has never been run yet
> -
>
> Key: AIRFLOW-6555
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6555
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.6
> Environment: localexec, mysql metastore, 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> # create a new dag
>  # go to graph view
>  # click on one of the tasks (it should have a white border)
>  # click on 'task instance details'
> below error appears
>  
> Traceback (most recent call last):
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 2446, in wsgi_app
>  response = self.full_dispatch_request()
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1951, in full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1820, in handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1949, in full_dispatch_request
>  rv = self.dispatch_request()
>  File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py", 
> line 1935, in dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
>  line 69, in inner
>  return self._run_view(f, *args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
>  line 368, in _run_view
>  return fn(self, *args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py",
>  line 261, in decorated_view
>  return func(*args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py", 
> line 295, in wrapper
>  return f(*args, **kwargs)
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py", 
> line 995, in task
>  dep_context=dep_context)]
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
>  line 643, in get_failed_dep_statuses
>  dep_context):
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py",
>  line 106, in get_dep_statuses
>  for dep_status in self._get_dep_statuses(ti, session, dep_context):
>  File 
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py",
>  line 50, in _get_dep_statuses
>  if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', 
> dagrun.run_id):
> AttributeError: 'NoneType' object has no attribute 'run_id'
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

codecov-io edited a comment on issue #7119: [AIRFLOW-5840] Add operator extra 
link to external task sensor
URL: https://github.com/apache/airflow/pull/7119#issuecomment-572737962
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=h1) 
Report
   > Merging 
[#7119](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/28d8ef1eff901cce36e6c3fa53b944a1cf7fa00a?src=pr=desc)
 will **decrease** coverage by `0.77%`.
   > The diff coverage is `90%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7119/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #7119  +/-   ##
   =
   - Coverage   85.17%   84.4%   -0.78% 
   =
 Files 683 710  +27 
 Lines   39155   39500 +345 
   =
   - Hits33352   33341  -11 
   - Misses   58036159 +356
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `90.03% <100%> (ø)` | :arrow_up: |
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `89.44% <60%> (-0.95%)` | :arrow_down: |
   | 
[airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5)
 | `88.04% <95.83%> (+2.51%)` | :arrow_up: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=)
 | `28.16% <0%> (-54.93%)` | :arrow_down: |
   | 
[...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5)
 | `50% <0%> (-50%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | ... and [98 
more](https://codecov.io/gh/apache/airflow/pull/7119/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=footer). 
Last update 
[28d8ef1...e0cab76](https://codecov.io/gh/apache/airflow/pull/7119?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator

[Kamil Bregula (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015377#comment-17015377
 ] 

Kamil Bregula commented on AIRFLOW-6546:


It seems that the cycles have already been blocked in the API. Hurra  Do you 
want to work on this ticket?

I have prepared a piece of code that can be helpful for you.

[https://pastebin.com/hDAiQPUZ]

If you want to write a copy operator, I'd write two methods in hook - 
list_files_recursive. It should also support filtering using wildcard. Another 
method can download a file to the specified directory. Operators can use this 
methods and NamedTemporaryFile to transfer files to GCS.

> add GDriveToGcsOperator
> ---
>
> Key: AIRFLOW-6546
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6546
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: lovk korm
>Priority: Major
>
> There is GcsToGDriveOperator but there isn't the equivalent in the other 
> direction
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stijndehaes opened a new pull request #7168: [AIRFLOW-6561] Add possibility to specify default resources for airfl…

[GitBox]

stijndehaes opened a new pull request #7168: [AIRFLOW-6561] Add possibility to 
specify default resources for airfl…
URL: https://github.com/apache/airflow/pull/7168
 
 
   …ow k8s workers
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6561) Kubernetes executor allow to set default resource requirements for created workers

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015363#comment-17015363
 ] 

ASF GitHub Bot commented on AIRFLOW-6561:
-

stijndehaes commented on pull request #7168: [AIRFLOW-6561] Add possibility to 
specify default resources for airfl…
URL: https://github.com/apache/airflow/pull/7168
 
 
   …ow k8s workers
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Kubernetes executor allow to set default resource requirements for created 
> workers
> --
>
> Key: AIRFLOW-6561
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6561
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executor-kubernetes
>Affects Versions: 1.10.7
>Reporter: Stijn De Haes
>Assignee: Stijn De Haes
>Priority: Major
>
> Allow setting default resource requirements for created airflow workers on 
> kubernetes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6561) Kubernetes executor allow to set default resource requirements for created workers

[Stijn De Haes (Jira)]

Stijn De Haes created AIRFLOW-6561:
--

 Summary: Kubernetes executor allow to set default resource 
requirements for created workers
 Key: AIRFLOW-6561
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6561
 Project: Apache Airflow
  Issue Type: Improvement
  Components: executor-kubernetes
Affects Versions: 1.10.7
Reporter: Stijn De Haes
Assignee: Stijn De Haes


Allow setting default resource requirements for created airflow workers on 
kubernetes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7164: [AIRFLOW-XXXX]: Set test env vars in confttest.py, not Breeze entrypoint

[GitBox]

potiuk merged pull request #7164: [AIRFLOW-]: Set test env vars in 
confttest.py, not Breeze entrypoint
URL: https://github.com/apache/airflow/pull/7164
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox]

potiuk commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error 
occurs when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#discussion_r366538449
 
 

 ##
 File path: .travis.yml
 ##
 @@ -87,6 +91,18 @@ jobs:
 BACKEND=sqlite
 PYTHON_VERSION=3.6
   stage: test
+- name: "Tests [Sqlite][3.7][multiprocessing(spawn)]"
+  env: >-
+AIRFLOW__CORE__MP_START_METHOD=spawn
 
 Review comment:
   No no - I mean to simply copy an existing test that runs one of test dags 
but to add this line:
   ```
   os.environment['AIRFLOW__CORE__MP_START_METHOD]='spawn'
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yqian1991 commented on a change in pull request #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules

[GitBox]

yqian1991 commented on a change in pull request #7167: [AIRFLOW-5974] AIP-21 
Change import paths for celery modules
URL: https://github.com/apache/airflow/pull/7167#discussion_r366531206
 
 

 ##
 File path: airflow/contrib/sensors/celery_queue_sensor.py
 ##
 @@ -16,74 +16,14 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""This module is deprecated. Please use 
`airflow.providers.apache.cassandra.hooks.cassandra`."""
 
 Review comment:
   stupid mistake, will fix it soon


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code

[GitBox]

codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=h1) 
Report
   > Merging 
[#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr=desc)
 will **increase** coverage by `0.69%`.
   > The diff coverage is `80.6%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6870/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6870  +/-   ##
   ==
   + Coverage   84.42%   85.12%   +0.69% 
   ==
 Files 679  710  +31 
 Lines   3851439488 +974 
   ==
   + Hits3251733615+1098 
   + Misses   5997 5873 -124
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5)
 | `52.67% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5)
 | `99.07% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=)
 | `89.28% <ø> (ø)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `100% <ø> (+22.39%)` | :arrow_up: |
   | 
[airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==)
 | `97.43% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `100% <ø> (+17.5%)` | :arrow_up: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `100% <ø> (+7.31%)` | :arrow_up: |
   | ... and [343 
more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=footer). 
Last update 
[4b7b977...3f8ec6f](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code

[GitBox]

codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=h1) 
Report
   > Merging 
[#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr=desc)
 will **increase** coverage by `0.23%`.
   > The diff coverage is `80.6%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6870/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6870  +/-   ##
   ==
   + Coverage   84.42%   84.66%   +0.23% 
   ==
 Files 679  710  +31 
 Lines   3851439488 +974 
   ==
   + Hits3251733432 +915 
   - Misses   5997 6056  +59
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5)
 | `52.67% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5)
 | `99.07% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=)
 | `89.28% <ø> (ø)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `100% <ø> (+22.39%)` | :arrow_up: |
   | 
[airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==)
 | `97.43% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `100% <ø> (+17.5%)` | :arrow_up: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `100% <ø> (+7.31%)` | :arrow_up: |
   | ... and [352 
more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=footer). 
Last update 
[4b7b977...3f8ec6f](https://codecov.io/gh/apache/airflow/pull/6870?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules

[GitBox]

mik-laj commented on a change in pull request #7167: [AIRFLOW-5974] AIP-21 
Change import paths for celery modules
URL: https://github.com/apache/airflow/pull/7167#discussion_r366525179
 
 

 ##
 File path: airflow/contrib/sensors/celery_queue_sensor.py
 ##
 @@ -16,74 +16,14 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""This module is deprecated. Please use 
`airflow.providers.apache.cassandra.hooks.cassandra`."""
 
 Review comment:
   ```suggestion
   """This module is deprecated. Please use 
`airflow.providers.celery.hooks.celery`."""
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

roitvt commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s 
operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#issuecomment-574327663
 
 
   > The `spark-on-k8s-operator` project is maintained by the Google Cloud 
Platform and should therefore comply with the GCP guidelines. I am not sure if 
it complies with these recommendations. Can you check it? 
https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit
   > 
   > In particular, system tests and guides are very much awaited. We have 
system tests for all operators for GCP. There are several services missing on 
the list of guides, but we are constantly developing.
   
   the spark-on-k8s-operator is being developed by GCP but it's not GCP 
service(can run on all Kubernetes flavors, not just GKE) so I don't know if it 
should follow GCP guidelines, or if it should and can be included in GCP system 
tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-5974) Change import paths for "celery" modules

[Yu Qian (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Qian reassigned AIRFLOW-5974:


Assignee: Yu Qian

> Change import paths for "celery" modules
> 
>
> Key: AIRFLOW-5974
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5974
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Yu Qian
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6560) db password leaks to logs

[Jira]

Marcin Jasiński created AIRFLOW-6560:


 Summary: db password leaks to logs
 Key: AIRFLOW-6560
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6560
 Project: Apache Airflow
  Issue Type: Bug
  Components: logging, security
Affects Versions: 1.10.6
Reporter: Marcin Jasiński


I have configured Airflow metadata db as MySQL.
{code:java}
sql_alchemy_conn = sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
After that I have used initdb command:
{code:java}
airflow initdb{code}
Tables in the airflow db have been created.

Then I have runed command:
{code:java}
airflow webserver -p 8080{code}
in logs:
{code:java}
[2020-01-14 18:39:03,457: INFO/MainProcess] Connected to 
sqla+mysql://airflow:airflow@localhost:3306/airflow{code}
password as a plain text.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5974) Change import paths for "celery" modules

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015316#comment-17015316
 ] 

ASF GitHub Bot commented on AIRFLOW-5974:
-

yqian1991 commented on pull request #7167: [AIRFLOW-5974] AIP-21 Change import 
paths for celery modules
URL: https://github.com/apache/airflow/pull/7167
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Change import paths for "celery" modules
> 
>
> Key: AIRFLOW-5974
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5974
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] yqian1991 opened a new pull request #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules

[GitBox]

yqian1991 opened a new pull request #7167: [AIRFLOW-5974] AIP-21 Change import 
paths for celery modules
URL: https://github.com/apache/airflow/pull/7167
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

[GitBox]

feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#discussion_r366514619
 
 

 ##
 File path: tests/contrib/hooks/test_aws_athena_hook.py
 ##
 @@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# 'License'); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+from unittest import mock
+
+try:
+from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook
+except ImportError:
+AWSAthenaHook = None  # type: ignore
+
+
+class MockAthenaClient:
 
 Review comment:
   Actually you only need to add `mock_conn.get_query_results.return_value = 
self.base_query_result` to the beginning of the test function. That is 
basically what you are already doing.
   
   You are doing good actually. But I would remove the extra class.
   This part
   ```
   if NextToken is None:
   return self.query_result_with_next
   else:
   return self.base_query_result
   ```
   is logic that you don't need if you set the 
`mock_conn.get_query_results.return_value` directly in the test function.
   
   A MockClass should not contain extra logic.
   
   Do you understand what I mean? :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

[GitBox]

feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#discussion_r366514619
 
 

 ##
 File path: tests/contrib/hooks/test_aws_athena_hook.py
 ##
 @@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# 'License'); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+from unittest import mock
+
+try:
+from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook
+except ImportError:
+AWSAthenaHook = None  # type: ignore
+
+
+class MockAthenaClient:
 
 Review comment:
   Actually you only need to add `mock_conn.get_query_results.return_value = 
self.base_query_result` to the beginning of the test function. That is 
basically what you are already doing.
   
   You are doing good actually. But I would remove the extra class.
   This part
   ```
   if NextToken is None:
   return self.query_result_with_next
   else:
   return self.base_query_result
   ```
   is logic that you don't need if you set the 
`mock_conn.get_query_results.return_value` directly in the test function.
   
   Do you understand what I mean? :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

roitvt commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366514865
 
 

 ##
 File path: airflow/contrib/sensors/spark_kubernetes_sensor.py
 ##
 @@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, Optional
+
+from kubernetes import client
+
+from airflow.contrib.hooks.kubernetes_hook import Kuberneteshook
+from airflow.exceptions import AirflowException
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesSensor(BaseSensorOperator):
+"""
+checks sparkapplication state on kubernetes
+:param sparkapplication_name: sparkapplication resource name
+:param namespace: the kubernetes namespace where the sparkApplication 
reside in
+:param kube_config: kubeconfig file location
+:param in_cluster: if airflow runs inside kubernetes pod take 
configuration from inside the cluster
+"""
+
+template_fields = ('sparkapplication_name', 'namespace', 'kube_config')
+INTERMEDIATE_STATES = ('SUBMITTED', 'RUNNING',)
+FAILURE_STATES = ('FAILED', 'SUBMISSION_FAILED', 'UNKNOWN')
+SUCCESS_STATES = 'COMPLETED'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_name: str,
+ namespace: str = 'default',
+ kube_config: Optional[str] = None,
+ in_cluster: bool = False,
+ *args, **kwargs):
+super().__init__(*args, **kwargs)
+self.sparkapplication_name = sparkapplication_name
+self.namespace = namespace
+self.kube_config = kube_config
+self.in_cluster = in_cluster
+self.api_client = Kuberneteshook(
 
 Review comment:
   I initialized the hook in the constructor so the poke method that can run 
many times in the sensor won't need to get a new connection each time it 
poke's. it's ok?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

roitvt commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366513445
 
 

 ##
 File path: airflow/contrib/operators/spark_kubernetes_operator.py
 ##
 @@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from kubernetes import client
+
+from airflow.contrib.hooks.kubernetes_hook import Kuberneteshook
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesOperator(BaseOperator):
+"""
+creates sparkapplication object in kubernetes cluster
+
+:param sparkapplication_object: kubernetes custom_resource_definition of 
sparkApplication
+:param namespace: kubernetes namespace to put sparkApplication
+:param kube_config: kubernetes kube_config path
+:param in_cluster: if airflow runs inside kubernetes pod take 
configuration from inside the cluster.
+"""
+
+template_fields = ['sparkapplication_object', 'namespace', 'kube_config']
+template_ext = ()
+ui_color = '#f4a460'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_object: dict,
+ namespace: str = 'default',
+ kube_config: Optional[str] = None,
+ in_cluster: bool = False,
+ *args, **kwargs) -> None:
+super().__init__(*args, **kwargs)
+self.sparkapplication_object = sparkapplication_object
+self.namespace = namespace
+self.kube_config = kube_config
+self.in_cluster = in_cluster
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use 
'BaseOperator.do_xcom_push' instead")
+
+def execute(self, context):
+self.log.info("creating sparkApplication")
+hook = Kuberneteshook(
+kube_config=self.kube_config,
+in_cluster=self.in_cluster
+)
+api_client = hook.get_conn()
+api = client.CustomObjectsApi(api_client)
 
 Review comment:
   the hook is creating the connection, the operator asks for the 
customObjectsApi if I need to do it in the hook I need to write code for each 
Kubernetes API object. I tried to make the Kubernetes hook general for various 
uses not just custom object api


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

[GitBox]

feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#discussion_r366509902
 
 

 ##
 File path: tests/contrib/hooks/test_aws_athena_hook.py
 ##
 @@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# 'License'); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+from unittest import mock
+
+try:
+from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook
+except ImportError:
+AWSAthenaHook = None  # type: ignore
+
+
+class MockAthenaClient:
 
 Review comment:
   You don't need the moto package to mock a class. Just mock the calls that 
are accessing the boto library in the get_query_results function. So that it 
will never really call the content of the moto get_query_results function.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6469: [AIRFLOW-5816] S3 to snowflake operator

[GitBox]

feluelle commented on a change in pull request #6469: [AIRFLOW-5816] S3 to 
snowflake operator
URL: https://github.com/apache/airflow/pull/6469#discussion_r366504190
 
 

 ##
 File path: airflow/providers/snowflake/operators/snowflake.py
 ##
 @@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.models import BaseOperator
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+from airflow.utils.decorators import apply_defaults
+
+
+class SnowflakeOperator(BaseOperator):
+"""
+Executes sql code in a Snowflake database
+
+:param snowflake_conn_id: reference to specific snowflake connection id
+:type snowflake_conn_id: str
+:param sql: the sql code to be executed. (templated)
+:type sql: Can receive a str representing a sql statement,
+a list of str (sql statements), or reference to a template file.
+Template reference are recognized by str ending in '.sql'
+:param autocommit: if True, each command is automatically committed.
+(default value: True)
+:type autocommit: bool
+:param parameters: (optional) the parameters to render the SQL query with.
+:type parameters: mapping or iterable
+:param warehouse: name of warehouse (will overwrite any warehouse
+defined in the connection's extra JSON)
+:type warehouse: str
+:param database: name of database (will overwrite database defined
+in connection)
+:type database: str
+:param schema: name of schema (will overwrite schema defined in
+connection)
+:type schema: str
+:param role: name of role (will overwrite any role defined in
+connection's extra JSON)
+:type role: str
+"""
+
+template_fields = ('sql',)
+template_ext = ('.sql',)
+ui_color = '#ededed'
+
+@apply_defaults
+def __init__(
+self, sql, snowflake_conn_id='snowflake_default', parameters=None,
+autocommit=True, warehouse=None, database=None, role=None,
+schema=None, *args, **kwargs):
+super(SnowflakeOperator, self).__init__(*args, **kwargs)
+self.snowflake_conn_id = snowflake_conn_id
+self.sql = sql
+self.autocommit = autocommit
+self.parameters = parameters
+self.warehouse = warehouse
+self.database = database
+self.role = role
+self.schema = schema
+
+def get_hook(self):
+"""
+Create and return SnowflakeHook.
+:return SnowflakeHook: An SnowflakeHook instance.
 
 Review comment:
   ```suggestion
   :return: a SnowflakeHook instance.
   :rtype: SnowflakeHook
   ```
   
   Then it should work to have an empty line inbetween.
   
   I think it is because of this being an invalid docstring `:return 
SnowflakeHook:`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7164: [AIRFLOW-XXXX]: Set test env vars in confttest.py, not Breeze entrypoint

[GitBox]

codecov-io commented on issue #7164: [AIRFLOW-]: Set test env vars in 
confttest.py, not Breeze entrypoint
URL: https://github.com/apache/airflow/pull/7164#issuecomment-574307599
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=h1) 
Report
   > Merging 
[#7164](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/086d731ce0066b3037d96df2a05cea1101ed3c17?src=pr=desc)
 will **decrease** coverage by `0.59%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7164/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=tree)
   
   ```diff
   @@   Coverage Diff@@
   ##   master   #7164 +/-   ##
   
   - Coverage   85.19%   84.6%   -0.6% 
   
 Files 710 710 
 Lines   39475   39479  +4 
   
   - Hits33631   33401-230 
   - Misses   58446078+234
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5)
 | `50% <0%> (-50%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/mongo\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvbW9uZ29fc2Vuc29yLnB5)
 | `53.33% <0%> (-46.67%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `56% <0%> (-44%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `49.65% <0%> (-38.78%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==)
 | `61.53% <0%> (-38.47%)` | :arrow_down: |
   | 
[airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=)
 | `28.75% <0%> (-23.75%)` | :arrow_down: |
   | ... and [16 
more](https://codecov.io/gh/apache/airflow/pull/7164/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=footer). 
Last update 
[086d731...875bdff](https://codecov.io/gh/apache/airflow/pull/7164?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] rconroy293 commented on a change in pull request #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

rconroy293 commented on a change in pull request #7119: [AIRFLOW-5840] Add 
operator extra link to external task sensor
URL: https://github.com/apache/airflow/pull/7119#discussion_r366496424
 
 

 ##
 File path: airflow/sensors/external_task_sensor.py
 ##
 @@ -16,22 +16,52 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
 import datetime
 import os
-from typing import Optional, Union
+from typing import FrozenSet, Optional, Union
 
 from sqlalchemy import func
 
+from airflow.configuration import conf
 from airflow.exceptions import AirflowException
-from airflow.models import DagBag, DagModel, DagRun, TaskInstance
+from airflow.models import BaseOperatorLink, DagBag, DagModel, DagRun, 
TaskInstance
 from airflow.operators.dummy_operator import DummyOperator
 from airflow.sensors.base_sensor_operator import BaseSensorOperator
 from airflow.utils.decorators import apply_defaults
 from airflow.utils.session import provide_session
 from airflow.utils.state import State
 
 
+def get_possible_target_execution_dates(execution_date, execution_delta, 
execution_date_fn):
+if execution_delta:
 
 Review comment:
   Added


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] rconroy293 commented on a change in pull request #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

rconroy293 commented on a change in pull request #7119: [AIRFLOW-5840] Add 
operator extra link to external task sensor
URL: https://github.com/apache/airflow/pull/7119#discussion_r366496493
 
 

 ##
 File path: airflow/plugins_manager.py
 ##
 @@ -133,6 +133,9 @@ def register_inbuilt_operator_links() -> None:
 except ImportError:
 pass
 
+from airflow.sensors.external_task_sensor import ExternalTaskLink
+inbuilt_operator_links.update([ExternalTaskLink])
+
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] rconroy293 commented on issue #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

rconroy293 commented on issue #7119: [AIRFLOW-5840] Add operator extra link to 
external task sensor
URL: https://github.com/apache/airflow/pull/7119#issuecomment-574305768
 
 
   Alright, I think thinks are in a good state @kaxil - things should be good 
on the serialization front. The failing build is labeled 
`kerberos-temporarily-disabled`, so I assume that's okay.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code

[GitBox]

lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check 
return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366495363
 
 

 ##
 File path: airflow/utils/state.py
 ##
 @@ -122,3 +122,10 @@ def unfinished(cls):
 cls.UP_FOR_RETRY,
 cls.UP_FOR_RESCHEDULE
 ]
+
+@classmethod
+def unsuccessful(cls):
+"""
+A list of states indicating that a task completed unsuccessfully.
+"""
+return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
 
 Review comment:
   @ashb Maybe it's because it's already resolved. Here is a screenshot
   
![image](https://user-images.githubusercontent.com/1295785/72370332-dee94a00-36b6-11ea-94c6-bafb5da91c90.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator

[Kamil Bregula (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015256#comment-17015256
 ] 

Kamil Bregula commented on AIRFLOW-6546:


I think that wildcard support is very helpful. I would like all operators that 
support the GCP/Google platform to have a similar set of features. This will 
facilitate their easier use. Maybe we should skip directories if we detect a 
cycle?

> add GDriveToGcsOperator
> ---
>
> Key: AIRFLOW-6546
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6546
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: lovk korm
>Priority: Major
>
> There is GcsToGDriveOperator but there isn't the equivalent in the other 
> direction
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s 
operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#issuecomment-574295974
 
 
   Have you tested it with GKE?  Should we ensure integration with IAM? 
KubernetesPodOperator have support for IAM via GKEStartPodOperator
   
https://github.com/apache/airflow/blob/6b1986ec58f10420dd9e397f91a388767a89b325/airflow/gcp/operators/kubernetes_engine.py#L185


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366484195
 
 

 ##
 File path: docs/operators-and-hooks-ref.rst
 ##
 @@ -1049,8 +1049,11 @@ These integrations allow you to perform various 
operations using various softwar
* - `Kubernetes `__
  - :doc:`How to use `
  -
+ - :mod:`airflow.contrib.hooks.kubernetes_hook`
 
 Review comment:
   This is probably not part of Kubernetes, but uses it, so it should be in a 
separate section. A similar situation is with Hadoop and Hive. Hive uses 
Hadoop, but Hive is not Hadoop.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366482784
 
 

 ##
 File path: airflow/contrib/operators/spark_kubernetes_operator.py
 ##
 @@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from kubernetes import client
+
+from airflow.contrib.hooks.kubernetes_hook import Kuberneteshook
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesOperator(BaseOperator):
+"""
+creates sparkapplication object in kubernetes cluster
+
+:param sparkapplication_object: kubernetes custom_resource_definition of 
sparkApplication
+:param namespace: kubernetes namespace to put sparkApplication
+:param kube_config: kubernetes kube_config path
+:param in_cluster: if airflow runs inside kubernetes pod take 
configuration from inside the cluster.
+"""
+
+template_fields = ['sparkapplication_object', 'namespace', 'kube_config']
+template_ext = ()
+ui_color = '#f4a460'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_object: dict,
+ namespace: str = 'default',
+ kube_config: Optional[str] = None,
+ in_cluster: bool = False,
+ *args, **kwargs) -> None:
+super().__init__(*args, **kwargs)
+self.sparkapplication_object = sparkapplication_object
+self.namespace = namespace
+self.kube_config = kube_config
+self.in_cluster = in_cluster
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use 
'BaseOperator.do_xcom_push' instead")
+
+def execute(self, context):
+self.log.info("creating sparkApplication")
+hook = Kuberneteshook(
+kube_config=self.kube_config,
+in_cluster=self.in_cluster
+)
+api_client = hook.get_conn()
+api = client.CustomObjectsApi(api_client)
 
 Review comment:
   It is not recommended to call methods from external libraries in the 
operator. This is the role of hook.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s 
operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#issuecomment-574293354
 
 
   The `spark-on-k8s-operator` project is maintained by the Google Cloud 
Platform and should therefore comply with the GCP guidelines. I am not sure if 
it complies with these recommendations. Can you check it?  
https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit
   
   In particular, system tests and guides are very much awaited. We have system 
tests for all operators for GCP. There are several services missing on the list 
of guides, but we are constantly developing.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366478778
 
 

 ##
 File path: airflow/contrib/sensors/spark_kubernetes_sensor.py
 ##
 @@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, Optional
+
+from kubernetes import client
+
+from airflow.contrib.hooks.kubernetes_hook import Kuberneteshook
+from airflow.exceptions import AirflowException
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesSensor(BaseSensorOperator):
+"""
+checks sparkapplication state on kubernetes
+:param sparkapplication_name: sparkapplication resource name
+:param namespace: the kubernetes namespace where the sparkApplication 
reside in
+:param kube_config: kubeconfig file location
+:param in_cluster: if airflow runs inside kubernetes pod take 
configuration from inside the cluster
+"""
+
+template_fields = ('sparkapplication_name', 'namespace', 'kube_config')
+INTERMEDIATE_STATES = ('SUBMITTED', 'RUNNING',)
+FAILURE_STATES = ('FAILED', 'SUBMISSION_FAILED', 'UNKNOWN')
+SUCCESS_STATES = 'COMPLETED'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_name: str,
+ namespace: str = 'default',
+ kube_config: Optional[str] = None,
+ in_cluster: bool = False,
+ *args, **kwargs):
+super().__init__(*args, **kwargs)
+self.sparkapplication_name = sparkapplication_name
+self.namespace = namespace
+self.kube_config = kube_config
+self.in_cluster = in_cluster
+self.api_client = Kuberneteshook(
 
 Review comment:
   Hook should not be initialized in the constructor but in the execute method. 
The constructor should be as light as possible, because this causes unnecessary 
load. DAG is loaded many times in various components, and the hook is needed 
only when executing the execute method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366477385
 
 

 ##
 File path: airflow/contrib/operators/spark_kubernetes_operator.py
 ##
 @@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from kubernetes import client
+
+from airflow.contrib.hooks.kubernetes_hook import Kuberneteshook
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesOperator(BaseOperator):
+"""
+creates sparkapplication object in kubernetes cluster
+
+:param sparkapplication_object: kubernetes custom_resource_definition of 
sparkApplication
+:param namespace: kubernetes namespace to put sparkApplication
+:param kube_config: kubernetes kube_config path
+:param in_cluster: if airflow runs inside kubernetes pod take 
configuration from inside the cluster.
+"""
+
+template_fields = ['sparkapplication_object', 'namespace', 'kube_config']
+template_ext = ()
+ui_color = '#f4a460'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_object: dict,
+ namespace: str = 'default',
+ kube_config: Optional[str] = None,
+ in_cluster: bool = False,
+ *args, **kwargs) -> None:
+super().__init__(*args, **kwargs)
+self.sparkapplication_object = sparkapplication_object
+self.namespace = namespace
+self.kube_config = kube_config
+self.in_cluster = in_cluster
+if kwargs.get('xcom_push') is not None:
 
 Review comment:
   ```suggestion
   ```
   It seems to me that it is not needed. Am I right? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

[GitBox]

mik-laj commented on a change in pull request #7163: [AIRFLOW-6542] add 
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r366476870
 
 

 ##
 File path: airflow/contrib/example_dags/example_spark_kubernetes_operator.py
 ##
 @@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import timedelta
+
+import yaml
+
+# [START import_module]
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+# Operators; we need this to operate!
+from airflow.contrib.operators.spark_kubernetes_operator import 
SparkKubernetesOperator
+from airflow.contrib.sensors.spark_kubernetes_sensor import 
SparkKubernetesSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+'owner': 'airflow',
+'depends_on_past': False,
+'start_date': days_ago(1),
+'email': ['airf...@example.com'],
+'email_on_failure': False,
+'email_on_retry': False,
+'max_active_runs': 1
+}
+# [END default_args]
+
+# [START instantiate_dag]
+spark_application_yaml = """
+#
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: "sparkoperator.k8s.io/v1beta2"
+kind: SparkApplication
+metadata:
+  name: spark-pi
+  namespace: default
+spec:
+  type: Scala
+  mode: cluster
+  image: "gcr.io/spark-operator/spark:v2.4.4"
+  imagePullPolicy: Always
+  mainClass: org.apache.spark.examples.SparkPi
+  mainApplicationFile: 
"local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar"
+  sparkVersion: "2.4.4"
+  restartPolicy:
+type: Never
+  volumes:
+- name: "test-volume"
+  hostPath:
+path: "/tmp"
+type: Directory
+  driver:
+cores: 1
+coreLimit: "1200m"
+memory: "512m"
+labels:
+  version: 2.4.4
+serviceAccount: default
+volumeMounts:
+  - name: "test-volume"
+mountPath: "/tmp"
+  executor:
+cores: 1
+instances: 1
+memory: "512m"
+labels:
+  version: 2.4.4
+volumeMounts:
+  - name: "test-volume"
+mountPath: "/tmp"
+"""
 
 Review comment:
   Can you store content inthe file and `template_ext`, please? DAG should be 
simple. 
   
https://airflow.readthedocs.io/en/latest/_api/airflow/models/baseoperator/index.html#airflow.models.baseoperator.BaseOperator.template_ext


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services