date:20181115

codecov-io edited a comment on issue #4165: [AIRFLOW-3322] Update qubole_hook 
to fetch command args dynamically from qds_sdk
URL: 
https://github.com/apache/incubator-airflow/pull/4165#issuecomment-437297736
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=h1)
 Report
   > Merging 
[#4165](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4165/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4165  +/-   ##
   =
   + Coverage   77.69%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16309  
   =
   + Hits12672   12673   +1 
   + Misses   36373636   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4165/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.37% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=footer).
 Last update 
[8668ef8...8ca1058](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4165: [AIRFLOW-3322] Update qubole_hook to fetch command args dynamically from qds_sdk

codecov-io edited a comment on issue #4165: [AIRFLOW-3322] Update qubole_hook 
to fetch command args dynamically from qds_sdk
URL: 
https://github.com/apache/incubator-airflow/pull/4165#issuecomment-437297736
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=h1)
 Report
   > Merging 
[#4165](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4165/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#4165   +/-   ##
   ===
 Coverage   77.69%   77.69%   
   ===
 Files 199  199   
 Lines   1630916309   
   ===
 Hits1267212672   
 Misses   3637 3637
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=footer).
 Last update 
[8668ef8...861860d](https://codecov.io/gh/apache/incubator-airflow/pull/4165?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4198: [AIRFLOW-3355] Fix BigQueryCursor.execute to work with Python3

codecov-io commented on issue #4198: [AIRFLOW-3355] Fix BigQueryCursor.execute 
to work with Python3
URL: 
https://github.com/apache/incubator-airflow/pull/4198#issuecomment-439291409
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=h1)
 Report
   > Merging 
[#4198](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4198/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4198  +/-   ##
   =
   + Coverage   77.69%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16309  
   =
   + Hits12672   12673   +1 
   + Misses   36373636   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4198/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.37% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=footer).
 Last update 
[8668ef8...eab19b0](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4198: [AIRFLOW-3355] Fix BigQueryCursor.execute to work with Python3

codecov-io edited a comment on issue #4198: [AIRFLOW-3355] Fix 
BigQueryCursor.execute to work with Python3
URL: 
https://github.com/apache/incubator-airflow/pull/4198#issuecomment-439291409
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=h1)
 Report
   > Merging 
[#4198](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4198/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4198  +/-   ##
   =
   + Coverage   77.69%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16309  
   =
   + Hits12672   12673   +1 
   + Misses   36373636   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4198/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.37% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=footer).
 Last update 
[8668ef8...eab19b0](https://codecov.io/gh/apache/incubator-airflow/pull/4198?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4190: Remove {table} parameter on COPY command

feng-tao commented on issue #4190: Remove {table} parameter on COPY command
URL: 
https://github.com/apache/incubator-airflow/pull/4190#issuecomment-439252062
 
 
   have you filed a jira ticket?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3355) Fix BigQueryCursor.execute to work with Python3



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688835#comment-16688835
 ] 

ASF GitHub Bot commented on AIRFLOW-3355:
-

sekikn opened a new pull request #4198: [AIRFLOW-3355] Fix 
BigQueryCursor.execute to work with Python3
URL: https://github.com/apache/incubator-airflow/pull/4198
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3355
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   BigQueryCursor.execute uses dict.iteritems internally,
   so it fails with Python3 if binding parameters are
   provided. This PR fixes this problem.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   tests.contrib.hooks.test_bigquery_hook:TestBigQueryCursor
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix BigQueryCursor.execute to work with Python3
> ---
>
> Key: AIRFLOW-3355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3355
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp, hooks
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
>
> {{BigQueryCursor.execute}} uses {{dict.iteritems}} internally, so it fails 
> with Python3 if binding parameters are provided.
> {code}
> In [1]: import sys
> In [2]: sys.version
> Out[2]: '3.6.6 (default, Sep 12 2018, 18:26:19) \n[GCC 8.0.1 20180414 
> (experimental) [trunk revision 259383]]'
> In [3]: from airflow.contrib.hooks.bigquery_hook import BigQueryHook
> In [4]: hook = BigQueryHook()
> In [5]: conn = hook.get_conn()
> [2018-11-15 19:01:35,856] {discovery.py:267} INFO - URL being requested: GET 
> https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
> In [6]: cur = conn.cursor()
> In [7]: cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})
> ---
> AttributeErrorTraceback (most recent call last)
>  in 
> > 1 cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})
> ~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in 
> execute(self, operation, parameters)
>1561 """
>1562 sql = _bind_parameters(operation,
> -> 1563parameters) if parameters else 
> operation
>1564 self.job_id = self.run_query(sql)
>1565
> ~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in 
> _bind_parameters(operation, parameters)
>1684 # inspired by MySQL Python Connector (conversion.py)
>1685 string_parameters = {}
> -> 1686 for (name, value) in parameters.iteritems():
>1687 if value is None:
>1688 string_parameters[name] = 'NULL'
> AttributeError: 'dict' object has no attribute 'iteritems'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] sekikn opened a new pull request #4198: [AIRFLOW-3355] Fix BigQueryCursor.execute to work with Python3

sekikn opened a new pull request #4198: [AIRFLOW-3355] Fix 
BigQueryCursor.execute to work with Python3
URL: https://github.com/apache/incubator-airflow/pull/4198
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3355
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   BigQueryCursor.execute uses dict.iteritems internally,
   so it fails with Python3 if binding parameters are
   provided. This PR fixes this problem.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   tests.contrib.hooks.test_bigquery_hook:TestBigQueryCursor
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3355) Fix BigQueryCursor.execute to work with Python3

2018-11-15 Thread Kengo Seki (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kengo Seki reassigned AIRFLOW-3355:
---

Assignee: Kengo Seki

> Fix BigQueryCursor.execute to work with Python3
> ---
>
> Key: AIRFLOW-3355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3355
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp, hooks
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
>
> {{BigQueryCursor.execute}} uses {{dict.iteritems}} internally, so it fails 
> with Python3 if binding parameters are provided.
> {code}
> In [1]: import sys
> In [2]: sys.version
> Out[2]: '3.6.6 (default, Sep 12 2018, 18:26:19) \n[GCC 8.0.1 20180414 
> (experimental) [trunk revision 259383]]'
> In [3]: from airflow.contrib.hooks.bigquery_hook import BigQueryHook
> In [4]: hook = BigQueryHook()
> In [5]: conn = hook.get_conn()
> [2018-11-15 19:01:35,856] {discovery.py:267} INFO - URL being requested: GET 
> https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
> In [6]: cur = conn.cursor()
> In [7]: cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})
> ---
> AttributeErrorTraceback (most recent call last)
>  in 
> > 1 cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})
> ~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in 
> execute(self, operation, parameters)
>1561 """
>1562 sql = _bind_parameters(operation,
> -> 1563parameters) if parameters else 
> operation
>1564 self.job_id = self.run_query(sql)
>1565
> ~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in 
> _bind_parameters(operation, parameters)
>1684 # inspired by MySQL Python Connector (conversion.py)
>1685 string_parameters = {}
> -> 1686 for (name, value) in parameters.iteritems():
>1687 if value is None:
>1688 string_parameters[name] = 'NULL'
> AttributeError: 'dict' object has no attribute 'iteritems'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4101: [AIRFLOW-3272] Add base grpc hook

codecov-io edited a comment on issue #4101: [AIRFLOW-3272] Add base grpc hook
URL: 
https://github.com/apache/incubator-airflow/pull/4101#issuecomment-433675239
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=h1)
 Report
   > Merging 
[#4101](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `60%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4101/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4101  +/-   ##
   ==
   - Coverage   77.69%   77.69%   -0.01% 
   ==
 Files 199  199  
 Lines   1630916312   +3 
   ==
   + Hits1267212674   +2 
   - Misses   3637 3638   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4101/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.35% <0%> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4101/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.32% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4101/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.29% <66.66%> (-0.04%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=footer).
 Last update 
[8668ef8...51dddc5](https://codecov.io/gh/apache/incubator-airflow/pull/4101?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4197: [AIRFLOW-3348] update run statistics on dag refresh

codecov-io edited a comment on issue #4197:  [AIRFLOW-3348] update run 
statistics on dag refresh
URL: 
https://github.com/apache/incubator-airflow/pull/4197#issuecomment-439221145
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=h1)
 Report
   > Merging 
[#4197](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4197/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4197  +/-   ##
   =
   + Coverage   77.69%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16310   +1 
   =
   + Hits12672   12674   +2 
   + Misses   36373636   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4197/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.34% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4197/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.37% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=footer).
 Last update 
[8668ef8...8a923ee](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4197: [AIRFLOW-3348] update run statistics on dag refresh

codecov-io commented on issue #4197:  [AIRFLOW-3348] update run statistics on 
dag refresh
URL: 
https://github.com/apache/incubator-airflow/pull/4197#issuecomment-439221145
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=h1)
 Report
   > Merging 
[#4197](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4197/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4197  +/-   ##
   =
   + Coverage   77.69%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16310   +1 
   =
   + Hits12672   12674   +2 
   + Misses   36373636   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4197/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.34% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4197/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.37% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=footer).
 Last update 
[8668ef8...8a923ee](https://codecov.io/gh/apache/incubator-airflow/pull/4197?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3348) Refresh run stats on dag refresh



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688723#comment-16688723
 ] 

ASF GitHub Bot commented on AIRFLOW-3348:
-

ms32035 opened a new pull request #4197:  [AIRFLOW-3348] update run statistics 
on dag refresh
URL: https://github.com/apache/incubator-airflow/pull/4197
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow Jira]
 - https://issues.apache.org/jira/browse/AIRFLOW-3348
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   In some cases dag run statistics may become outdated, for example, when dag 
run is deleted.
   
   As the view from which dag runs are deleted permits to delete runs from 
multiple dags in a single transaction, this seems to be the most reasonable 
place to update run statistics
   
   ### Tests
   
   - [X] My PR does not need testing for this extremely good reason:
Covered by existing test for `/refresh`
   
   ### Commits
   
   - [X ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Refresh run stats on dag refresh
> 
>
> Key: AIRFLOW-3348
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3348
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Marcin Szymanski
>Assignee: Marcin Szymanski
>Priority: Minor
>
> In some cases dag run statistics may become outdated, for example, when dag 
> run is deleted.
> As the view from which dag runs are deleted permits to delete runs from 
> multiple dags in a single transaction, this seems to be the most reasonable 
> place to update run statistics



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ms32035 opened a new pull request #4197: [AIRFLOW-3348] update run statistics on dag refresh

ms32035 opened a new pull request #4197:  [AIRFLOW-3348] update run statistics 
on dag refresh
URL: https://github.com/apache/incubator-airflow/pull/4197
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow Jira]
 - https://issues.apache.org/jira/browse/AIRFLOW-3348
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   In some cases dag run statistics may become outdated, for example, when dag 
run is deleted.
   
   As the view from which dag runs are deleted permits to delete runs from 
multiple dags in a single transaction, this seems to be the most reasonable 
place to update run statistics
   
   ### Tests
   
   - [X] My PR does not need testing for this extremely good reason:
Covered by existing test for `/refresh`
   
   ### Commits
   
   - [X ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3357) Scheduler doesn't work on example DAGs unless there's some dag file to process

2018-11-15 Thread JIRA

Victor Villas Bôas Chaves created AIRFLOW-3357:
--

 Summary: Scheduler doesn't work on example DAGs unless there's 
some dag file to process
 Key: AIRFLOW-3357
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3357
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 1.10.0, 1.10.1
Reporter: Victor Villas Bôas Chaves


Having a blank Airflow install, if you try to manually run one of the example 
DAGs no tasks are going to get queued or executed. They're going to stay with 
state null.

Steps to reproduce on a new airflow:
 # Entered the UI, turned on the example_bash_operator, manually triggered the 
example_bash_operator, 6 tasks went to None state, nothing gets scheduled
 # Rebooted the scheduler (with debug logging on). Nothing gets scheduled.
 # Create a mytutorial.py in the dag folder with code from tutorial.py but DAG 
name changed, everything starts getting scheduled.

A debug view of the logs is here: 
[https://gist.github.com/tomfaulhaber/4e72ed0884c9580c606e02e4b745ddff]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] rmn36 commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

rmn36 commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will 
consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439204626
 
 
   @xnuinside thanks! rebased and re-ran tests. All green now!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

codecov-io edited a comment on issue #4182: [AIRFLOW-3336] Add new TriggerRule 
that will consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439204135
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=h1)
 Report
   > Merging 
[#4182](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `60%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4182  +/-   ##
   ==
   - Coverage   77.69%   77.68%   -0.02% 
   ==
 Files 199  199  
 Lines   1630916319  +10 
   ==
   + Hits1267212678   +6 
   - Misses   3637 3641   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.32% <ø> (ø)` | :arrow_up: |
   | 
[airflow/utils/trigger\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90cmlnZ2VyX3J1bGUucHk=)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==)
 | `90.14% <55.55%> (-5.03%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=footer).
 Last update 
[8668ef8...dfacbf6](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

codecov-io commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that 
will consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439204135
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=h1)
 Report
   > Merging 
[#4182](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8668ef869d3d844dac746ec88609d3710a1264ab?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `60%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4182  +/-   ##
   ==
   - Coverage   77.69%   77.68%   -0.02% 
   ==
 Files 199  199  
 Lines   1630916319  +10 
   ==
   + Hits1267212678   +6 
   - Misses   3637 3641   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `92.32% <ø> (ø)` | :arrow_up: |
   | 
[airflow/utils/trigger\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90cmlnZ2VyX3J1bGUucHk=)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/incubator-airflow/pull/4182/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==)
 | `90.14% <55.55%> (-5.03%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=footer).
 Last update 
[8668ef8...dfacbf6](https://codecov.io/gh/apache/incubator-airflow/pull/4182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside edited a comment on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

xnuinside edited a comment on issue #4182: [AIRFLOW-3336] Add new TriggerRule 
that will consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439189896
 
 
   @rmn36, we fix today issue with Redis versions, what was the reason of tests 
fails - you can re-trigger your build, it must be okay.
   
   and +1 for PR
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

xnuinside commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that 
will consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439189896
 
 
   @rmn36, we fix today issue with Redis versions, what was the reason of tests 
fails - you can re-trigger your build, it must be okay.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439180998
 
 
   @ashb, all tests are passed, need I to add more tests?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3350) docs - explain how to use bitshift with lists

2018-11-15 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-3350.
-
Resolution: Fixed

Resolved by https://github.com/apache/incubator-airflow/pull/4191

> docs - explain how to use bitshift with lists
> -
>
> Key: AIRFLOW-3350
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3350
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 1.10.0
>Reporter: jack
>Assignee: jack
>Priority: Trivial
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3350) docs - explain how to use bitshift with lists



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688609#comment-16688609
 ] 

ASF GitHub Bot commented on AIRFLOW-3350:
-

kaxil closed pull request #4191: [AIRFLOW-3350] Explain how to use Bitshift 
Composition with lists
URL: https://github.com/apache/incubator-airflow/pull/4191
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/concepts.rst b/docs/concepts.rst
index 7c0603efa4..2896010248 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -240,6 +240,25 @@ We can put this all together to build a simple pipeline:
 python_callable=lambda: print("GOODBYE!"))
 )
 
+Bitshift can also be used with lists. For example:
+
+.. code:: python
+
+op1 >> [op2, op3]
+
+is equivalent to:
+
+.. code:: python
+
+op1 >> op2
+op1 >> op3
+
+and equivalent to:
+
+.. code:: python
+
+op1.set_downstream([op2, op3])
+
 Tasks
 =
 
@@ -908,4 +927,3 @@ would not be scanned by Airflow at all. This improves 
efficiency of DAG finding)
 The scope of a ``.airflowignore`` file is the directory it is in plus all its 
subfolders.
 You can also prepare ``.airflowignore`` file for a subfolder in ``DAG_FOLDER`` 
and it
 would only be applicable for that subfolder.
-


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> docs - explain how to use bitshift with lists
> -
>
> Key: AIRFLOW-3350
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3350
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 1.10.0
>Reporter: jack
>Assignee: jack
>Priority: Trivial
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil closed pull request #4191: [AIRFLOW-3350] Explain how to use Bitshift Composition with lists

kaxil closed pull request #4191: [AIRFLOW-3350] Explain how to use Bitshift 
Composition with lists
URL: https://github.com/apache/incubator-airflow/pull/4191
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/concepts.rst b/docs/concepts.rst
index 7c0603efa4..2896010248 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -240,6 +240,25 @@ We can put this all together to build a simple pipeline:
 python_callable=lambda: print("GOODBYE!"))
 )
 
+Bitshift can also be used with lists. For example:
+
+.. code:: python
+
+op1 >> [op2, op3]
+
+is equivalent to:
+
+.. code:: python
+
+op1 >> op2
+op1 >> op3
+
+and equivalent to:
+
+.. code:: python
+
+op1.set_downstream([op2, op3])
+
 Tasks
 =
 
@@ -908,4 +927,3 @@ would not be scanned by Airflow at all. This improves 
efficiency of DAG finding)
 The scope of a ``.airflowignore`` file is the directory it is in plus all its 
subfolders.
 You can also prepare ``.airflowignore`` file for a subfolder in ``DAG_FOLDER`` 
and it
 would only be applicable for that subfolder.
-


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3345) Add Google Cloud Storage (GCS) operators for ACL

2018-11-15 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-3345.
-
   Resolution: Fixed
Fix Version/s: 1.10.2
   2.0.0

Resolved by https://github.com/apache/incubator-airflow/pull/4192

> Add Google Cloud Storage (GCS) operators for ACL
> 
>
> Key: AIRFLOW-3345
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3345
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: gcp
>Reporter: Szymon Przedwojski
>Assignee: Szymon Przedwojski
>Priority: Minor
> Fix For: 2.0.0, 1.10.2
>
>
> Add operators to be able to add ACL entries to GCS buckets and objects:
>  * 
> [https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert]
>  * 
> [https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3345) Add Google Cloud Storage (GCS) operators for ACL



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688604#comment-16688604
 ] 

ASF GitHub Bot commented on AIRFLOW-3345:
-

kaxil closed pull request #4192: [AIRFLOW-3345] Add Google Cloud Storage (GCS) 
operators for ACL
URL: https://github.com/apache/incubator-airflow/pull/4192
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/example_dags/example_gcs_acl.py 
b/airflow/contrib/example_dags/example_gcs_acl.py
new file mode 100644
index 00..7247199a4f
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcs_acl.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates a new ACL entry on the specified bucket and 
object.
+
+This DAG relies on the following OS environment variables
+
+* GCS_ACL_BUCKET - Name of a bucket.
+* GCS_ACL_OBJECT - Name of the object. For information about how to URL encode 
object
+names to be path safe, see:
+https://cloud.google.com/storage/docs/json_api/#encoding
+* GCS_ACL_ENTITY - The entity holding the permission.
+* GCS_ACL_BUCKET_ROLE - The access permission for the entity for the bucket.
+* GCS_ACL_OBJECT_ROLE - The access permission for the entity for the object.
+"""
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcs_acl_operator import \
+GoogleCloudStorageBucketCreateAclEntryOperator, \
+GoogleCloudStorageObjectCreateAclEntryOperator
+
+# [START howto_operator_gcs_acl_args_common]
+GCS_ACL_BUCKET = os.environ.get('GCS_ACL_BUCKET', 'example-bucket')
+GCS_ACL_OBJECT = os.environ.get('GCS_ACL_OBJECT', 'example-object')
+GCS_ACL_ENTITY = os.environ.get('GCS_ACL_ENTITY', 'example-entity')
+GCS_ACL_BUCKET_ROLE = os.environ.get('GCS_ACL_BUCKET_ROLE', 
'example-bucket-role')
+GCS_ACL_OBJECT_ROLE = os.environ.get('GCS_ACL_OBJECT_ROLE', 
'example-object-role')
+# [END howto_operator_gcs_acl_args_common]
+
+default_args = {
+'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+'example_gcs_acl',
+default_args=default_args,
+schedule_interval=None  # Change to match your use case
+) as dag:
+# [START howto_operator_gcs_bucket_create_acl_entry_task]
+gcs_bucket_create_acl_entry_task = 
GoogleCloudStorageBucketCreateAclEntryOperator(
+bucket=GCS_ACL_BUCKET,
+entity=GCS_ACL_ENTITY,
+role=GCS_ACL_BUCKET_ROLE,
+task_id="gcs_bucket_create_acl_entry_task"
+)
+# [END howto_operator_gcs_bucket_create_acl_entry_task]
+# [START howto_operator_gcs_object_create_acl_entry_task]
+gcs_object_create_acl_entry_task = 
GoogleCloudStorageObjectCreateAclEntryOperator(
+bucket=GCS_ACL_BUCKET,
+object_name=GCS_ACL_OBJECT,
+entity=GCS_ACL_ENTITY,
+role=GCS_ACL_OBJECT_ROLE,
+task_id="gcs_object_create_acl_entry_task"
+)
+# [END howto_operator_gcs_object_create_acl_entry_task]
+
+gcs_bucket_create_acl_entry_task >> gcs_object_create_acl_entry_task
diff --git a/airflow/contrib/hooks/gcs_hook.py 
b/airflow/contrib/hooks/gcs_hook.py
index dc92b0cb3e..f848d25dce 100644
--- a/airflow/contrib/hooks/gcs_hook.py
+++ b/airflow/contrib/hooks/gcs_hook.py
@@ -568,6 +568,104 @@ def create_bucket(self,
 'Bucket creation failed. Error was: {}'.format(ex.content)
 )
 
+def insert_bucket_acl(self, bucket, entity, role, user_project):
+# type: (str, str, str, str) -> None
+"""
+Creates a new ACL entry on the specified bucket.
+See: 
https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert
+
+:param bucket: Name of a bucket.
+:type bucket: str
+:param entity: The entity holding the permission, in one of the 
following forms:
+- user-userId
+- user-email
+- group-groupId
+- group-email
+

[GitHub] kaxil commented on a change in pull request #4192: [AIRFLOW-3345] Add Google Cloud Storage (GCS) operators for ACL

kaxil commented on a change in pull request #4192: [AIRFLOW-3345] Add Google 
Cloud Storage (GCS) operators for ACL
URL: https://github.com/apache/incubator-airflow/pull/4192#discussion_r233995078
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -568,6 +568,104 @@ def create_bucket(self,
 'Bucket creation failed. Error was: {}'.format(ex.content)
 )
 
+def insert_bucket_acl(self, bucket, entity, role, user_project):
+# type: (str, str, str, str) -> None
+"""
+Creates a new ACL entry on the specified bucket.
+See: 
https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert
+
+:param bucket: Name of a bucket.
+:type bucket: str
+:param entity: The entity holding the permission, in one of the 
following forms:
+- user-userId
 
 Review comment:
   Needs indentation


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil closed pull request #4192: [AIRFLOW-3345] Add Google Cloud Storage (GCS) operators for ACL

kaxil closed pull request #4192: [AIRFLOW-3345] Add Google Cloud Storage (GCS) 
operators for ACL
URL: https://github.com/apache/incubator-airflow/pull/4192
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/example_dags/example_gcs_acl.py 
b/airflow/contrib/example_dags/example_gcs_acl.py
new file mode 100644
index 00..7247199a4f
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcs_acl.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates a new ACL entry on the specified bucket and 
object.
+
+This DAG relies on the following OS environment variables
+
+* GCS_ACL_BUCKET - Name of a bucket.
+* GCS_ACL_OBJECT - Name of the object. For information about how to URL encode 
object
+names to be path safe, see:
+https://cloud.google.com/storage/docs/json_api/#encoding
+* GCS_ACL_ENTITY - The entity holding the permission.
+* GCS_ACL_BUCKET_ROLE - The access permission for the entity for the bucket.
+* GCS_ACL_OBJECT_ROLE - The access permission for the entity for the object.
+"""
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcs_acl_operator import \
+GoogleCloudStorageBucketCreateAclEntryOperator, \
+GoogleCloudStorageObjectCreateAclEntryOperator
+
+# [START howto_operator_gcs_acl_args_common]
+GCS_ACL_BUCKET = os.environ.get('GCS_ACL_BUCKET', 'example-bucket')
+GCS_ACL_OBJECT = os.environ.get('GCS_ACL_OBJECT', 'example-object')
+GCS_ACL_ENTITY = os.environ.get('GCS_ACL_ENTITY', 'example-entity')
+GCS_ACL_BUCKET_ROLE = os.environ.get('GCS_ACL_BUCKET_ROLE', 
'example-bucket-role')
+GCS_ACL_OBJECT_ROLE = os.environ.get('GCS_ACL_OBJECT_ROLE', 
'example-object-role')
+# [END howto_operator_gcs_acl_args_common]
+
+default_args = {
+'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+'example_gcs_acl',
+default_args=default_args,
+schedule_interval=None  # Change to match your use case
+) as dag:
+# [START howto_operator_gcs_bucket_create_acl_entry_task]
+gcs_bucket_create_acl_entry_task = 
GoogleCloudStorageBucketCreateAclEntryOperator(
+bucket=GCS_ACL_BUCKET,
+entity=GCS_ACL_ENTITY,
+role=GCS_ACL_BUCKET_ROLE,
+task_id="gcs_bucket_create_acl_entry_task"
+)
+# [END howto_operator_gcs_bucket_create_acl_entry_task]
+# [START howto_operator_gcs_object_create_acl_entry_task]
+gcs_object_create_acl_entry_task = 
GoogleCloudStorageObjectCreateAclEntryOperator(
+bucket=GCS_ACL_BUCKET,
+object_name=GCS_ACL_OBJECT,
+entity=GCS_ACL_ENTITY,
+role=GCS_ACL_OBJECT_ROLE,
+task_id="gcs_object_create_acl_entry_task"
+)
+# [END howto_operator_gcs_object_create_acl_entry_task]
+
+gcs_bucket_create_acl_entry_task >> gcs_object_create_acl_entry_task
diff --git a/airflow/contrib/hooks/gcs_hook.py 
b/airflow/contrib/hooks/gcs_hook.py
index dc92b0cb3e..f848d25dce 100644
--- a/airflow/contrib/hooks/gcs_hook.py
+++ b/airflow/contrib/hooks/gcs_hook.py
@@ -568,6 +568,104 @@ def create_bucket(self,
 'Bucket creation failed. Error was: {}'.format(ex.content)
 )
 
+def insert_bucket_acl(self, bucket, entity, role, user_project):
+# type: (str, str, str, str) -> None
+"""
+Creates a new ACL entry on the specified bucket.
+See: 
https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert
+
+:param bucket: Name of a bucket.
+:type bucket: str
+:param entity: The entity holding the permission, in one of the 
following forms:
+- user-userId
+- user-email
+- group-groupId
+- group-email
+- domain-domain
+- project-team-projectId
+- allUsers
+- allAuthenticatedUsers
+:type entity: str
+:param role: The access permission for the entity.
+Acceptable values are: "OWNER",

[GitHub] feluelle commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

feluelle commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will 
consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439172815
 
 
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4189: [AIRFLOW-3346] Add hook and operator for GCP transfer service.

codecov-io commented on issue #4189: [AIRFLOW-3346] Add hook and operator for 
GCP transfer service.
URL: 
https://github.com/apache/incubator-airflow/pull/4189#issuecomment-439172483
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=h1)
 Report
   > Merging 
[#4189](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8c8e41f4e46e0ecaa545b1435f7ccf3a76bcfd3b?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4189/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #4189   +/-   ##
   ==
 Coverage77.7%   77.7%   
   ==
 Files 199 199   
 Lines   16309   16309   
   ==
 Hits12673   12673   
 Misses   36363636
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=footer).
 Last update 
[8c8e41f...869431b](https://codecov.io/gh/apache/incubator-airflow/pull/4189?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3356) Scheduler gets stuck for certain DAGs

2018-11-15 Thread John Smodic (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Smodic updated AIRFLOW-3356:
-
Description: 
I observe the scheduler getting stuck for certain DAGs:

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
Runtime Last Run

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s 
2018-11-15T13:38:47

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 2018-11-15T19:11:44

 

The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
what it's doing. There's no mention of that DAG in the scheduler logs otherwise.

 

The mapped process looks like this:

ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
/usr/local/bin/airflow scheduler

 

  was:
I observe the scheduler getting stuck for certain DAGs:

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
Runtime Last Run

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s 
2018-11-15T13:38:47

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 2018-11-15T19:11:44

 

The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
what it's doing. There's no mention of that DAG in the scheduler logs otherwise.

 

The mapped process looks like this:

ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
/usr/local/bin/airflow scheduler


> Scheduler gets stuck for certain DAGs
> -
>
> Key: AIRFLOW-3356
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3356
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Smodic
>Priority: Critical
>
> I observe the scheduler getting stuck for certain DAGs:
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
> Runtime Last Run
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
> /home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s 
> 2018-11-15T13:38:47
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
> /home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 
> 2018-11-15T19:11:44
>  
> The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
> what it's doing. There's no mention of that DAG in the scheduler logs 
> otherwise.
>  
> The mapped process looks like this:
> ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
> /usr/local/bin/airflow scheduler
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3356) Scheduler gets stuck for certain DAGs

2018-11-15 Thread John Smodic (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Smodic updated AIRFLOW-3356:
-
Description: 
I observe the scheduler getting stuck for certain DAGs:

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
Runtime Last Run

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s 
2018-11-15T13:38:47

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 2018-11-15T19:11:44

 

The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
what it's doing. There's no mention of that DAG in the scheduler logs otherwise.

 

The mapped process looks like this:

ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
/usr/local/bin/airflow scheduler

  was:
I observe the scheduler getting stuck for certain DAGs:

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
Runtime Last Run

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/stuck_dag.py 14241 19977.55s 1.05s 2018-11-15T13:38:47

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 2018-11-15T19:11:44

 

The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
what it's doing. There's no mention of that DAG in the scheduler logs otherwise.

 

The mapped process looks like this:

ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
/usr/local/bin/airflow scheduler


> Scheduler gets stuck for certain DAGs
> -
>
> Key: AIRFLOW-3356
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3356
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Smodic
>Priority: Critical
>
> I observe the scheduler getting stuck for certain DAGs:
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
> Runtime Last Run
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
> /home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s 
> 2018-11-15T13:38:47
> Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
> /home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 
> 2018-11-15T19:11:44
>  
> The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
> what it's doing. There's no mention of that DAG in the scheduler logs 
> otherwise.
>  
> The mapped process looks like this:
> ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
> /usr/local/bin/airflow scheduler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3356) Scheduler gets stuck for certain DAGs

2018-11-15 Thread John Smodic (JIRA)

John Smodic created AIRFLOW-3356:


 Summary: Scheduler gets stuck for certain DAGs
 Key: AIRFLOW-3356
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3356
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 1.9.0
Reporter: John Smodic


I observe the scheduler getting stuck for certain DAGs:

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last 
Runtime Last Run

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/stuck_dag.py 14241 19977.55s 1.05s 2018-11-15T13:38:47

Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: 
/home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s 2018-11-15T19:11:44

 

The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell 
what it's doing. There's no mention of that DAG in the scheduler logs otherwise.

 

The mapped process looks like this:

ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 
/usr/local/bin/airflow scheduler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3355) Fix BigQueryCursor.execute to work with Python3

2018-11-15 Thread Kengo Seki (JIRA)

Kengo Seki created AIRFLOW-3355:
---

 Summary: Fix BigQueryCursor.execute to work with Python3
 Key: AIRFLOW-3355
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3355
 Project: Apache Airflow
  Issue Type: Bug
  Components: gcp, hooks
Reporter: Kengo Seki


{{BigQueryCursor.execute}} uses {{dict.iteritems}} internally, so it fails with 
Python3 if binding parameters are provided.

{code}
In [1]: import sys

In [2]: sys.version
Out[2]: '3.6.6 (default, Sep 12 2018, 18:26:19) \n[GCC 8.0.1 20180414 
(experimental) [trunk revision 259383]]'

In [3]: from airflow.contrib.hooks.bigquery_hook import BigQueryHook

In [4]: hook = BigQueryHook()

In [5]: conn = hook.get_conn()
[2018-11-15 19:01:35,856] {discovery.py:267} INFO - URL being requested: GET 
https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest

In [6]: cur = conn.cursor()

In [7]: cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})
---
AttributeErrorTraceback (most recent call last)
 in 
> 1 cur.execute("SELECT count(*) FROM ds.t WHERE c = %(v)d", {"v": 0})

~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in execute(self, 
operation, parameters)
   1561 """
   1562 sql = _bind_parameters(operation,
-> 1563parameters) if parameters else operation
   1564 self.job_id = self.run_query(sql)
   1565

~/dev/incubator-airflow/airflow/contrib/hooks/bigquery_hook.py in 
_bind_parameters(operation, parameters)
   1684 # inspired by MySQL Python Connector (conversion.py)
   1685 string_parameters = {}
-> 1686 for (name, value) in parameters.iteritems():
   1687 if value is None:
   1688 string_parameters[name] = 'NULL'

AttributeError: 'dict' object has no attribute 'iteritems'
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] andrewhharmon commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

andrewhharmon commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that 
will consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439145730
 
 
   I have been looking for this functionality as well. Hope it gets merged 
soon. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2680) Don't automatically percolate skipped state

2018-11-15 Thread Andrew Harmon (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688469#comment-16688469
 ] 

Andrew Harmon commented on AIRFLOW-2680:


I'm looking for a solution to the same issue. I don't think you always want the 
SKIP status to propagate downstream. It would be nice to have some more control 
of this

> Don't automatically percolate skipped state
> ---
>
> Key: AIRFLOW-2680
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2680
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Andrei-Alin Popescu
>Assignee: Andrei-Alin Popescu
>Priority: Major
>
> Dear Airflow Maintainers,
>  
> As part of our workflow, we have cases where all the upstream of a certain 
> task A can be skipped. In this case, airflow seems to automatically mark A as 
> skipped.
> However, this does not quite work for us, since there are changes external to 
> the DAG which A needs to process, regardless of whether its upstream ran or 
> not. Additionally, we require A to get into an "upstream_failed" state and 
> not run if any its upstream tasks failed.
> I don't see a trigger rule to cover this, so what would be the best way to 
> achieve this? I was thinking we could attach a DummyOperator as an upstream 
> to A, which in a way marks the fact that A depends on some external data and 
> needs to run anyway, but this can get really ugly for big DAGs.
> I was also thinking we could have a new trigger_rule, e.g. "no_failure", 
> which would only trigger tasks if no upstream has failed. It differs from 
> "all_success" in that it will also trigger if all upstream has been skipped, 
> rather than percolating the skipped state on.
> I'd really appreciate your feedback on this, and I'd like to know if in fact 
> there is already a good way of doing this with airflow that I don't know of.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] seelmann commented on issue #4195: [AIRFLOW-3353] Pin redis version

seelmann commented on issue #4195: [AIRFLOW-3353] Pin redis version
URL: 
https://github.com/apache/incubator-airflow/pull/4195#issuecomment-439139684
 
 
   @xnuinside @ashb thanks for the fast fix


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439129336
 
 
   @ashb 
   the second question about args - yes, you can mix it and all work ok, if you 
want - I can add mix args test case, it's not a big deal
   
   but you need to use --do-handshake-on-connect without =true, because if you 
use this flag it's == true - 
http://docs.gunicorn.org/en/stable/settings.html#do-handshake-on-connect 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439128812
 
 
   @ashb, I know about this feature, it works from env variable by default. We 
don't need to do something for the support it, but a user should set up env 
variables, what not always acceptable


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3353.

   Resolution: Fixed
Fix Version/s: 2.0.0

> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 2.0.0
>
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439129336
 
 
   @ashb 
   the second question about args - yes, you can mix it and all work ok, if you 
want - I can add mix args test case, it's not a big deal


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-3353 started by Ash Berlin-Taylor.
--
> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 2.0.0
>
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439128812
 
 
   @ashb, I know about this feature, it works from env variable by default. We 
don't need to do something for the support it, but a user should set up env 
variables, what not always acceptable
   
   the second question about args - yes, you can mix it and all work ok, if you 
want - I can add mix args test case, it's not a big deal


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439128812
 
 
   @ashb, I know about this feature, it works from env variable by default. We 
don't need to do something for the support it, but a user should set up env 
variables, what not always acceptable


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config 
param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439128602
 
 
   If we do have this, I was expecting the usage to be like:
   
   ```sh
   airflow webserver --do-handshake-on-connect=true --graceful-timeout 60 -w 2
   ```
   
   I.e. being able to freely mix our args with Gunicorn's directly. Is this how 
your PR makes it work or not?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3352) Don't Show Airflow config in rbac view based on flag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Phanindhra updated AIRFLOW-3352:

Fix Version/s: 1.10.2

> Don't Show Airflow config in rbac view based on flag
> 
>
> Key: AIRFLOW-3352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3352
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.0
>Reporter: Sai Phanindhra
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: configuration, rbac, webserver
> Fix For: 1.10.2
>
>
> Earlier expose_config flags is used to toggle whether to show configuration 
> in UI or not. This feature is not enabled when rbac is enabled. Add provision 
> to toggle this feature when rbac is enabled on airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3352) Don't Show Airflow config in rbac view based on flag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Phanindhra updated AIRFLOW-3352:

Labels: configuration rbac webserver  (was: rbac)

> Don't Show Airflow config in rbac view based on flag
> 
>
> Key: AIRFLOW-3352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3352
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.0
>Reporter: Sai Phanindhra
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: configuration, rbac, webserver
> Fix For: 1.10.2
>
>
> Earlier expose_config flags is used to toggle whether to show configuration 
> in UI or not. This feature is not enabled when rbac is enabled. Add provision 
> to toggle this feature when rbac is enabled on airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config 
param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439127715
 
 
   Oh, I just found this in the Gunicorn docs 
http://docs.gunicorn.org/en/stable/settings.html#settings
   
   > Settings can be specified by using environment variable GUNICORN_CMD_ARGS. 
All available command line arguments can be used. For example, to specify the 
bind address and number of workers:
   > 
   > ```sh
   > $ GUNICORN_CMD_ARGS="--bind=127.0.0.1 --workers=3" gunicorn app:app
   > ```
   
   Do we need special code in Airflow to handle this, or is it worth just 
adding someething to our docs to mention this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3352) Don't Show Airflow config in rbac view based on flag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Phanindhra updated AIRFLOW-3352:

Component/s: webserver

> Don't Show Airflow config in rbac view based on flag
> 
>
> Key: AIRFLOW-3352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3352
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.0
>Reporter: Sai Phanindhra
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: rbac
>
> Earlier expose_config flags is used to toggle whether to show configuration 
> in UI or not. This feature is not enabled when rbac is enabled. Add provision 
> to toggle this feature when rbac is enabled on airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3352) Don't Show Airflow config in rbac view based on flag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Phanindhra updated AIRFLOW-3352:

Affects Version/s: 1.10.0

> Don't Show Airflow config in rbac view based on flag
> 
>
> Key: AIRFLOW-3352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3352
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.0
>Reporter: Sai Phanindhra
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: rbac
>
> Earlier expose_config flags is used to toggle whether to show configuration 
> in UI or not. This feature is not enabled when rbac is enabled. Add provision 
> to toggle this feature when rbac is enabled on airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (AIRFLOW-3352) Don't Show Airflow config in rbac view based on flag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-3352 started by Sai Phanindhra.
---
> Don't Show Airflow config in rbac view based on flag
> 
>
> Key: AIRFLOW-3352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3352
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Sai Phanindhra
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: rbac
>
> Earlier expose_config flags is used to toggle whether to show configuration 
> in UI or not. This feature is not enabled when rbac is enabled. Add provision 
> to toggle this feature when rbac is enabled on airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] tedmiston commented on issue #3656: [AIRFLOW-2803] Fix all ESLint issues

tedmiston commented on issue #3656: [AIRFLOW-2803] Fix all ESLint issues
URL: 
https://github.com/apache/incubator-airflow/pull/3656#issuecomment-439126242
 
 
   Hey @jmcarp thanks for bumping this... it looks like I forgot to close it 
out.  I think it's done shy of a rebase but I may have minor local changes on 
my machine.  I've blocked time tonight to dig into it again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688411#comment-16688411
 ] 

ASF GitHub Bot commented on AIRFLOW-3353:
-

ashb closed pull request #4195: [AIRFLOW-3353] Pin redis version
URL: https://github.com/apache/incubator-airflow/pull/4195
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/setup.py b/setup.py
index 8c6c927153..fa56c1b9a4 100644
--- a/setup.py
+++ b/setup.py
@@ -218,7 +218,7 @@ def write_version(filename=os.path.join(*['airflow',
 postgres = ['psycopg2-binary>=2.7.4']
 qds = ['qds-sdk>=1.9.6']
 rabbitmq = ['librabbitmq>=1.6.1']
-redis = ['redis>=2.10.5']
+redis = ['redis>=2.10.5,<3.0.0']
 s3 = ['boto3>=1.7.0, <1.8.0']
 salesforce = ['simple-salesforce>=0.72']
 samba = ['pysmbclient>=0.1.3']


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Iuliia Volkova
>Priority: Major
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb closed pull request #4195: [AIRFLOW-3353] Pin redis version

ashb closed pull request #4195: [AIRFLOW-3353] Pin redis version
URL: https://github.com/apache/incubator-airflow/pull/4195
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/setup.py b/setup.py
index 8c6c927153..fa56c1b9a4 100644
--- a/setup.py
+++ b/setup.py
@@ -218,7 +218,7 @@ def write_version(filename=os.path.join(*['airflow',
 postgres = ['psycopg2-binary>=2.7.4']
 qds = ['qds-sdk>=1.9.6']
 rabbitmq = ['librabbitmq>=1.6.1']
-redis = ['redis>=2.10.5']
+redis = ['redis>=2.10.5,<3.0.0']
 s3 = ['boto3>=1.7.0, <1.8.0']
 salesforce = ['simple-salesforce>=0.72']
 samba = ['pysmbclient>=0.1.3']


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4195: [AIRFLOW-3353] Pin redis version

xnuinside commented on issue #4195: [AIRFLOW-3353] Pin redis version
URL: 
https://github.com/apache/incubator-airflow/pull/4195#issuecomment-439124521
 
 
   @ashb, tests are passed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4195: [AIRFLOW-3353] Pin redis version

codecov-io commented on issue #4195: [AIRFLOW-3353] Pin redis version
URL: 
https://github.com/apache/incubator-airflow/pull/4195#issuecomment-439124322
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=h1)
 Report
   > Merging 
[#4195](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/d5035c1ee9bfd292cdad7800ff9f8363c5d08a49?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4195/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #4195   +/-   ##
   ==
 Coverage77.7%   77.7%   
   ==
 Files 199 199   
 Lines   16309   16309   
   ==
 Hits12673   12673   
 Misses   36363636
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=footer).
 Last update 
[d5035c1...1bba9c9](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4195: [AIRFLOW-3353] Pin redis version

codecov-io edited a comment on issue #4195: [AIRFLOW-3353] Pin redis version
URL: 
https://github.com/apache/incubator-airflow/pull/4195#issuecomment-439124322
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=h1)
 Report
   > Merging 
[#4195](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/d5035c1ee9bfd292cdad7800ff9f8363c5d08a49?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4195/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #4195   +/-   ##
   ==
 Coverage77.7%   77.7%   
   ==
 Files 199 199   
 Lines   16309   16309   
   ==
 Hits12673   12673   
 Misses   36363636
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=footer).
 Last update 
[d5035c1...1bba9c9](https://codecov.io/gh/apache/incubator-airflow/pull/4195?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] potiuk commented on issue #4196: Support for cloud build

potiuk commented on issue #4196: Support for cloud build
URL: 
https://github.com/apache/incubator-airflow/pull/4196#issuecomment-439123468
 
 
   Again PR by mistake. Sorry :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 commented on issue #4111: [AIRFLOW-3266] Add AWS Athena Operator and hook

phani8996 commented on issue #4111: [AIRFLOW-3266] Add AWS Athena Operator and 
hook
URL: 
https://github.com/apache/incubator-airflow/pull/4111#issuecomment-439123362
 
 
   @ashb @kaxil Can you please review these final changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] potiuk opened a new pull request #4196: Support for cloud build

potiuk opened a new pull request #4196: Support for cloud build
URL: https://github.com/apache/incubator-airflow/pull/4196
 
 
   Make sure you have checked _all_ steps below.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] potiuk closed pull request #4196: Support for cloud build

potiuk closed pull request #4196: Support for cloud build
URL: https://github.com/apache/incubator-airflow/pull/4196
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/example_dags/example_gcp_compute_igm.py 
b/airflow/contrib/example_dags/example_gcp_compute_igm.py
index 3e4543c60d..718b813aad 100644
--- a/airflow/contrib/example_dags/example_gcp_compute_igm.py
+++ b/airflow/contrib/example_dags/example_gcp_compute_igm.py
@@ -76,13 +76,13 @@
 
 SOURCE_TEMPLATE_URL = os.environ.get(
 'SOURCE_TEMPLATE_URL',
-"https://www.googleapis.com/compute/beta/projects/;
-"example-project/global/instanceTemplates/instance-template-test")
+"https://www.googleapis.com/compute/beta/projects/; + PROJECT_ID +
+"/global/instanceTemplates/instance-template-test")
 
 DESTINATION_TEMPLATE_URL = os.environ.get(
 'DESTINATION_TEMPLATE_URL',
-"https://www.googleapis.com/compute/beta/projects/;
-"example-airflow/global/instanceTemplates/" + NEW_TEMPLATE_NAME)
+"https://www.googleapis.com/compute/beta/projects/; + PROJECT_ID +
+"/global/instanceTemplates/" + NEW_TEMPLATE_NAME)
 
 UPDATE_POLICY = {
 "type": "OPPORTUNISTIC",
diff --git a/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py 
b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
index 76563d7596..6c1ad89818 100644
--- a/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
+++ b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
@@ -53,7 +53,7 @@
 SOURCE_REPOSITORY = os.environ.get(
 'SOURCE_REPOSITORY',
 'https://source.developers.google.com/'
-'projects/example-project/repos/hello-world/moveable-aliases/master')
+'projects/{}/repos/hello-world/moveable-aliases/master'.format(PROJECT_ID))
 ZIP_PATH = os.environ.get('ZIP_PATH', '')
 ENTRYPOINT = os.environ.get('ENTRYPOINT', 'helloWorld')
 FUNCTION_NAME = 'projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, 
LOCATION,
diff --git a/airflow/contrib/example_dags/example_gcp_sql_query.py 
b/airflow/contrib/example_dags/example_gcp_sql_query.py
index af7c5c2447..f4f80526e4 100644
--- a/airflow/contrib/example_dags/example_gcp_sql_query.py
+++ b/airflow/contrib/example_dags/example_gcp_sql_query.py
@@ -42,6 +42,7 @@
 
 import os
 import subprocess
+from os.path import expanduser
 
 from six.moves.urllib.parse import quote_plus
 
@@ -61,11 +62,11 @@
 POSTGRES_PUBLIC_IP = os.environ.get('POSTGRES_PUBLIC_IP', '0.0.0.0')
 POSTGRES_PUBLIC_PORT = os.environ.get('POSTGRES_PUBLIC_PORT', 5432)
 POSTGRES_CLIENT_CERT_FILE = os.environ.get('POSTGRES_CLIENT_CERT_FILE',
-   "/tmp/client-cert.pem")
+   ".key/client-cert.pem")
 POSTGRES_CLIENT_KEY_FILE = os.environ.get('POSTGRES_CLIENT_KEY_FILE',
-  "/tmp/client-key.pem")
+  ".key/client-key.pem")
 POSTGRES_SERVER_CA_FILE = os.environ.get('POSTGRES_SERVER_CA_FILE',
- "/tmp/server-ca.pem")
+ ".key/server-ca.pem")
 
 MYSQL_INSTANCE_NAME = os.environ.get('MYSQL_INSTANCE_NAME', 'testmysql')
 MYSQL_DATABASE_NAME = os.environ.get('MYSQL_DATABASE_NAME', 'mysqldb')
@@ -74,11 +75,11 @@
 MYSQL_PUBLIC_IP = os.environ.get('MYSQL_PUBLIC_IP', '0.0.0.0')
 MYSQL_PUBLIC_PORT = os.environ.get('MYSQL_PUBLIC_PORT', 3306)
 MYSQL_CLIENT_CERT_FILE = os.environ.get('MYSQL_CLIENT_CERT_FILE',
-"/tmp/client-cert.pem")
+".key/client-cert.pem")
 MYSQL_CLIENT_KEY_FILE = os.environ.get('MYSQL_CLIENT_KEY_FILE',
-   "/tmp/client-key.pem")
+   ".key/client-key.pem")
 MYSQL_SERVER_CA_FILE = os.environ.get('MYSQL_SERVER_CA_FILE',
-  "/tmp/server-ca.pem")
+  ".key/server-ca.pem")
 
 SQL = [
 'CREATE TABLE IF NOT EXISTS TABLE_TEST (I INTEGER)',
@@ -97,6 +98,16 @@
 
 # [START howto_operator_cloudsql_query_connections]
 
+HOME_DIR = expanduser("~")
+
+
+def get_absolute_path(path):
+if path.startswith("/"):
+return path
+else:
+return os.path.join(HOME_DIR, path)
+
+
 postgres_kwargs = dict(
 user=quote_plus(POSTGRES_USER),
 password=quote_plus(POSTGRES_PASSWORD),
@@ -106,9 +117,9 @@
 location=quote_plus(LOCATION),
 instance=quote_plus(POSTGRES_INSTANCE_NAME),
 database=quote_plus(POSTGRES_DATABASE_NAME),
-client_cert_file=quote_plus(POSTGRES_CLIENT_CERT_FILE),
-client_key_file=quote_plus(POSTGRES_CLIENT_KEY_FILE),
-

[GitHub] codecov-io edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

codecov-io edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-437659189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=h1)
 Report
   > Merging 
[#4174](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/d5035c1ee9bfd292cdad7800ff9f8363c5d08a49?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `53.94%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4174/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4174  +/-   ##
   =
   + Coverage77.7%   77.7%   +<.01% 
   =
 Files 199 199  
 Lines   16309   16319  +10 
   =
   + Hits12673   12681   +8 
   - Misses   36363638   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4174/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `64.77% <53.94%> (+0.18%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=footer).
 Last update 
[d5035c1...b75ec80](https://codecov.io/gh/apache/incubator-airflow/pull/4174?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439116184
 
 
   @ashb, all passed 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439114161
 
 
   @ashb, test_get_conn (tests.contrib.hooks.test_redis_hook.TestRedisHook) ... 
passed in current run, seems it works ) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688374#comment-16688374
 ] 

ASF GitHub Bot commented on AIRFLOW-3353:
-

xnuinside opened a new pull request #4195: [AIRFLOW-3353] Pin redis version
URL: https://github.com/apache/incubator-airflow/pull/4195
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Iuliia Volkova
>Priority: Major
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
>

[jira] [Commented] (AIRFLOW-2722) ECSOperator requires network configuration parameter when FARGATE launch type is used

2018-11-15 Thread Craig Forster (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688372#comment-16688372
 ] 

Craig Forster commented on AIRFLOW-2722:


This might have been already fixed under AIRFLOW-3074

> ECSOperator requires network configuration parameter when FARGATE launch type 
> is used
> -
>
> Key: AIRFLOW-2722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2722
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 2.0.0
>Reporter: Craig Forster
>Priority: Major
>
> The 'FARGATE' launch type was added in AIRFLOW-2435, however when using that 
> launch mode the following error is returned:
> {noformat}
> Network Configuration must be provided when networkMode 'awsvpc' is specified.
> {noformat}
> Fargate-launched tasks use the "awsvpc" networking type, and as per the 
> [boto3 
> documentation|http://boto3.readthedocs.io/en/latest/reference/services/ecs.html#ECS.Client.run_task]
>  for run_task:
> {quote}This parameter is required for task definitions that use the awsvpc 
> network mode to receive their own Elastic Network Interface, and it is not 
> supported for other network modes.
> {quote}
> As it's currently implemented, the Fargate launch type is unusable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside opened a new pull request #4195: [AIRFLOW-3353] Pin redis version

xnuinside opened a new pull request #4195: [AIRFLOW-3353] Pin redis version
URL: https://github.com/apache/incubator-airflow/pull/4195
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor

2018-11-15 Thread Iuliia Volkova (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iuliia Volkova reassigned AIRFLOW-3353:
---

Assignee: Iuliia Volkova

> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Iuliia Volkova
>Priority: Major
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config 
param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439110409
 
 
   If it works that should probably be a separate PR - there's alredy a JIRA 
for that opened today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3344) Airflow DAG object clear function does not clear tasks in the upstream_failed state when only_failed=True

2018-11-15 Thread Steve Jacobs (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Jacobs updated AIRFLOW-3344:
--
Description: 
When using the airflow clear command from the cli, you can pass an argument 
--only_failed to clear only failed tasks. This will clear ONLY tasks with the 
state failed, and not tasks with the state upstream_failed, causing any clear 
to still fail the dag_run if any upstream tasks are failed.

Since one_failed as a trigger rule also checks for upstream_failed tasks, it 
seems consistent that this should also clear upstream_failed tasks. The 
relevant code change necessary is here:
{code:java}
if only_failed:
 tis = tis.filter(TI.state == State.FAILED)
{code}
to
{code:java}
if only_failed:
  tis = tis.filter(TI.state.in_([State.FAILED, State.UPSTREAM_FAILED]))
{code}
in models.py

Additionally when clearing dags, the dag_run is set to the running state, but 
the dag_run start_date is not updated to the current time, as it is when 
clearing tasks through the Web UI. This causes dag_runs to fail on their 
timeouts even if the dag is full of successful tasks. This needs to be changed 
as well.

  was:
When using the airflow clear command from the cli, you can pass an argument 
--only_failed to clear only failed tasks. This will clear ONLY tasks with the 
state failed, and not tasks with the state upstream_failed, causing any clear 
to still fail the dag_run if any upstream tasks are failed.

Since one_failed as a trigger rule also checks for upstream_failed tasks, it 
seems consistent that this should also clear upstream_failed tasks. The 
relevant code change necessary is here:
{code:java}
if only_failed:
 tis = tis.filter(TI.state == State.FAILED)
{code}
to
{code:java}
if only_failed:
  tis = tis.filter(TI.state.in_([State.FAILED, State.UPSTREAM_FAILED]))
{code}
in models.py


> Airflow DAG object clear function does not clear tasks in the upstream_failed 
> state when only_failed=True
> -
>
> Key: AIRFLOW-3344
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3344
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.8.2, 1.9.0, 1.10.0
>Reporter: Steve Jacobs
>Priority: Minor
>  Labels: easyfix, newbie
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When using the airflow clear command from the cli, you can pass an argument 
> --only_failed to clear only failed tasks. This will clear ONLY tasks with the 
> state failed, and not tasks with the state upstream_failed, causing any clear 
> to still fail the dag_run if any upstream tasks are failed.
> Since one_failed as a trigger rule also checks for upstream_failed tasks, it 
> seems consistent that this should also clear upstream_failed tasks. The 
> relevant code change necessary is here:
> {code:java}
> if only_failed:
>  tis = tis.filter(TI.state == State.FAILED)
> {code}
> to
> {code:java}
> if only_failed:
>   tis = tis.filter(TI.state.in_([State.FAILED, State.UPSTREAM_FAILED]))
> {code}
> in models.py
> Additionally when clearing dags, the dag_run is set to the running state, but 
> the dag_run start_date is not updated to the current time, as it is when 
> clearing tasks through the Web UI. This causes dag_runs to fail on their 
> timeouts even if the dag is full of successful tasks. This needs to be 
> changed as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] rmn36 commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will consider skipped ancestors as success

rmn36 commented on issue #4182: [AIRFLOW-3336] Add new TriggerRule that will 
consider skipped ancestors as success 
URL: 
https://github.com/apache/incubator-airflow/pull/4182#issuecomment-439105033
 
 
   Failing test is
   `FAIL: test_get_conn (tests.contrib.hooks.test_redis_hook.TestRedisHook)`
   which is unrelated to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439103191
 
 
   @ashb , I added version pin, wait for Travis now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439099206
 
 
   Please!
   
   See also https://github.com/celery/kombu/pull/946


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config 
param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439099206
 
 
   Please!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3354) Scheduler compares offset-naive and offset-aware dates



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688250#comment-16688250
 ] 

Ash Berlin-Taylor commented on AIRFLOW-3354:


Might be enough if you provide all the ways that `start_date` is set in your 
DAG/tasks

> Scheduler compares offset-naive and offset-aware dates
> --
>
> Key: AIRFLOW-3354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3354
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10.1
>Reporter: Jakub Powierza
>Priority: Major
>
> New version of Airflow (1.10.1rc1 and 1.10.1rc2) tries to compare 
> offset-naive and offset-aware dates in Scheduler. I've tested a simple case 
> with schedule set to "*/10 * * * *". I've tried to clean my developer 
> instance with `airflow resetdb` and start from scratch but it does not help 
> at all. This issue does not occur on stable version 1.10.0.
> My setup: Python 3.6 on Ubuntu 14.04 with Airflow Scheduler based on Celery 
> with RabbitMQ backend.
> Exception found in Scheduler logs:
> {code:java}
> 2018-11-15 14:41:23,194:ERROR:airflow.processor:[CT=None] Got an exception! 
> Propagating...
> Traceback (most recent call last):
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> Process DagFileProcessor40-Process:
> Traceback (most recent call last):
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
>  self.run()
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
>  self._target(*self._args, **self._kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] smentek commented on issue #4090: [AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls

smentek commented on issue #4090: [AIRFLOW-3250] Fix for Redis Hook for not 
authorised connection calls
URL: 
https://github.com/apache/incubator-airflow/pull/4090#issuecomment-439087875
 
 
   OK I'll check it. Would it go on current PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3354) Scheduler compares offset-naive and offset-aware dates

2018-11-15 Thread Jakub Powierza (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688244#comment-16688244
 ] 

Jakub Powierza commented on AIRFLOW-3354:
-

Sadly not due to confidential use but I'll prepare a reproduction today or 
tomorrow and post it here.

> Scheduler compares offset-naive and offset-aware dates
> --
>
> Key: AIRFLOW-3354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3354
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10.1
>Reporter: Jakub Powierza
>Priority: Major
>
> New version of Airflow (1.10.1rc1 and 1.10.1rc2) tries to compare 
> offset-naive and offset-aware dates in Scheduler. I've tested a simple case 
> with schedule set to "*/10 * * * *". I've tried to clean my developer 
> instance with `airflow resetdb` and start from scratch but it does not help 
> at all. This issue does not occur on stable version 1.10.0.
> My setup: Python 3.6 on Ubuntu 14.04 with Airflow Scheduler based on Celery 
> with RabbitMQ backend.
> Exception found in Scheduler logs:
> {code:java}
> 2018-11-15 14:41:23,194:ERROR:airflow.processor:[CT=None] Got an exception! 
> Propagating...
> Traceback (most recent call last):
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> Process DagFileProcessor40-Process:
> Traceback (most recent call last):
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
>  self.run()
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
>  self._target(*self._args, **self._kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439081682
 
 
   @ashb, for now, maybe set up dependency with 
https://www.python.org/dev/peps/pep-0440/#compatible-release '~=' to fix it 
quick and then investigate what's new with 3.0.0? Can I help with it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439081682
 
 
   @ashb, maybe set up now dependency with 
https://www.python.org/dev/peps/pep-0440/#compatible-release '~=' to fix it 
quick and then investigate what's new with 3.0.0? Can I help with it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config 
param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439077370
 
 
   redis-py 3.0.0 just released a new version and it broke some things :(
   
   In this particular case it just broke the tests, but not anything beyond 
that. But it might :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside commented on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439073558
 
 
   @ashb , can you look whats going on with Travis or ping somebody? One test 
fails on several PRs - [fail] 0.00% 
tests.contrib.hooks.test_redis_hook.TestRedisHook.test_get_conn: 0.0227s same 
issue here - https://travis-ci.org/apache/incubator-airflow/jobs/455404767 (pr 
with doc's changes) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function

xnuinside edited a comment on issue #4174: [AIRFLOW-571] Airflow CLI: add 
gunicorn_config param and refactor webserver cli function
URL: 
https://github.com/apache/incubator-airflow/pull/4174#issuecomment-439073558
 
 
   @ashb , can you look whats going on with Travis or ping somebody? One test 
fails on several PRs - [fail] 0.00% 
tests.contrib.hooks.test_redis_hook.TestRedisHook.test_get_conn: 0.0227s same 
issue here - https://travis-ci.org/apache/incubator-airflow/jobs/455404767 (pr 
with doc's changes) , here also this test one of failed 
https://travis-ci.org/apache/incubator-airflow/builds/455525726?utm_source=github_status_medium=notification
 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3354) Scheduler compares offset-naive and offset-aware dates



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688185#comment-16688185
 ] 

Ash Berlin-Taylor commented on AIRFLOW-3354:


Can you provide the full dag definition?

> Scheduler compares offset-naive and offset-aware dates
> --
>
> Key: AIRFLOW-3354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3354
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10.1
>Reporter: Jakub Powierza
>Priority: Major
>
> New version of Airflow (1.10.1rc1 and 1.10.1rc2) tries to compare 
> offset-naive and offset-aware dates in Scheduler. I've tested a simple case 
> with schedule set to "*/10 * * * *". I've tried to clean my developer 
> instance with `airflow resetdb` and start from scratch but it does not help 
> at all. This issue does not occur on stable version 1.10.0.
> My setup: Python 3.6 on Ubuntu 14.04 with Airflow Scheduler based on Celery 
> with RabbitMQ backend.
> Exception found in Scheduler logs:
> {code:java}
> 2018-11-15 14:41:23,194:ERROR:airflow.processor:[CT=None] Got an exception! 
> Propagating...
> Traceback (most recent call last):
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> Process DagFileProcessor40-Process:
> Traceback (most recent call last):
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
>  self.run()
>  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
>  self._target(*self._args, **self._kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 389, in helper
>  pickle_dags)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1846, in process_file
>  self._process_dags(dagbag, dags, ti_keys_to_schedule)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 1426, in _process_dags
>  dag_run = self.create_dag_run(dag)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/jobs.py", 
> line 909, in create_dag_run
>  external_trigger=False
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 74, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 4270, in create_dagrun
>  run.verify_integrity(session=session)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 70, in wrapper
>  return func(*args, **kwargs)
>  File 
> "/home/jpowierz/my_project/venv/lib/python3.6/site-packages/airflow/models.py",
>  line 5215, in verify_integrity
>  if task.start_date > self.execution_date and not self.is_backfill:
> TypeError: can't compare offset-naive and offset-aware datetimes
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…

codecov-io edited a comment on issue #3683: [AIRFLOW-2770] kubernetes: add 
support for dag folder in the docker i…
URL: 
https://github.com/apache/incubator-airflow/pull/3683#issuecomment-409915837
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=h1)
 Report
   > Merging 
[#3683](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/e703d6beeb379ee88ef5e7df495e8a785666f8af?src=pr=desc)
 will **increase** coverage by `0.86%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3683/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3683  +/-   ##
   ==
   + Coverage   76.67%   77.53%   +0.86% 
   ==
 Files 199  205   +6 
 Lines   1618615771 -415 
   ==
   - Hits1241012228 -182 
   + Misses   3776 3543 -233
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/slack\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc2xhY2tfb3BlcmF0b3IucHk=)
 | `0% <0%> (-97.37%)` | :arrow_down: |
   | 
[airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=)
 | `31.03% <0%> (-68.97%)` | :arrow_down: |
   | 
[airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=)
 | `41.17% <0%> (-58.83%)` | :arrow_down: |
   | 
[airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5)
 | `67.07% <0%> (-17.31%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=)
 | `78.94% <0%> (-15.79%)` | :arrow_down: |
   | 
[airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5)
 | `78% <0%> (-12%)` | :arrow_down: |
   | 
[airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=)
 | `90.47% <0%> (-9.53%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `73.91% <0%> (-7.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `83.95% <0%> (-5.47%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.58% <0%> (-3.47%)` | :arrow_down: |
   | ... and [95 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=footer).
 Last update 
[e703d6b...9401914](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688102#comment-16688102
 ] 

Ash Berlin-Taylor commented on AIRFLOW-3353:


This is maybe a problem in celery/kombu, and not Airflow itself?

> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Priority: Major
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688097#comment-16688097
 ] 

Ash Berlin-Taylor commented on AIRFLOW-3353:


Could you check if redis==3.0.0post1 also has this problem?

> redis-py 3.0.0 dependency breaks celery executor
> 
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Priority: Major
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
> 205, in start
> self.blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 369, in start
> return self.obj.start()
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 317, in start
> blueprint.start(self)
>   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 
> 119, in start
> step.start(parent)
>   File 
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
> line 593, in start
> c.loop(*c.loop_args())
>   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
> 91, in asynloop
> next(loop)
>   File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", 
> line 354, in create_loop
> cb(*cbargs)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 337, in on_readable
> chan.handlers[type]()
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 983, in _deliver
> callback(message)
>   File 
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", 
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
>   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", 
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
>   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, 
> in zadd
> for pair in iteritems(mapping):
>   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, 
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg H closed AIRFLOW-3182.
---
Resolution: Feedback Received

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png, refactored.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1419) Trigger Rule not respected downstream of BranchPythonOperator

2018-11-15 Thread Iuliia Volkova (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688037#comment-16688037
 ] 

Iuliia Volkova commented on AIRFLOW-1419:
-

[~conradlee], without dummy task, you don't have a branch, you just have the 
task confluence_op what depend on branch_op, it means confluence_op  is a 
branch by itself, it not depend on some branch, it's branch. We don't have in 
Airflow edges what you can say - o, this edge is Branch without a task. 

In your case on your picture, confluence_op - is a branch of branch operator, 
what never be returned by branch operator and it also depends on the result of 
another branch. 

> Trigger Rule not respected downstream of BranchPythonOperator
> -
>
> Key: AIRFLOW-1419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1419
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Conrad Lee
>Priority: Major
>
> Lets consider the following DAG:
> {noformat}
>   
>  /  \
> branch_op confluence_op
>  \__work_op/
> {noformat}
> This is implemented in the following code:
> {code:java}
> import airflow
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.utils.trigger_rule import TriggerRule
> from airflow.models import DAG
> args = {
> 'owner': 'airflow',
> 'start_date': airflow.utils.dates.days_ago(2)
> }
> dag = DAG(
> dag_id='branch_skip_problem',
> default_args=args,
> schedule_interval="@daily")
> branch_op = BranchPythonOperator(
> task_id='branch_op',
> python_callable=lambda: 'work_op',
> dag=dag)
> work_op = DummyOperator(task_id='work_op', dag=dag)
> confluence_op = DummyOperator(task_id='confluence_op', dag=dag, 
> trigger_rule=TriggerRule.ALL_DONE)
> branch_op.set_downstream(confluence_op)
> branch_op.set_downstream(work_op)
> work_op.set_downstream(confluence_op)
> {code}
> Note that branch_op is a BranchPythonOperator, work_op and confluence_op are 
> DummyOperators, and that confluence_op has its trigger_rule set to ALL_DONE.
> In dag runs where brancher_op chooses to activate work_op as its child, 
> confluence_op never runs. This doesn't seem right, because confluence_op has 
> two parents and a trigger_rule set that it'll run as soon as all of its 
> parents are done (whether or not they are skipped).
> I know this example seems contrived and that in practice there are better 
> ways of conditionally executing work_op. However, this is the minimal code to 
> illustrate the problem. You can imagine that this problem might actually 
> creep up in practice where originally there was a good reason to use the 
> BranchPythonOperator, and then time passes and someone modifies one of the 
> branches so that it doesn't really contain any children anymore, thus 
> resembling the example.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg H updated AIRFLOW-3182:

Attachment: refactored.png

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png, refactored.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688015#comment-16688015
 ] 

Greg H edited comment on AIRFLOW-3182 at 11/15/18 1:12 PM:
---

OK sorry, I missed your previous comment :) In fact I already did it with 
another branch after each hour and it works fine now. The previous behaviour 
was just not logical to me and wanted to report it. Thanks for your help.

This is the refactored one:

!refactored.png!


was (Author: zeckt):
OK sorry, I missed your previous comment :) In fact I already did it with 
another branch after each hour and it works fine now. The previous behaviour 
was just not logical to me and wanted to report it. Thanks for your help.

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png, refactored.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688015#comment-16688015
 ] 

Greg H commented on AIRFLOW-3182:
-

OK sorry, I missed your previous comment :) In fact I already did it with 
another branch after each hour and it works fine now. The previous behaviour 
was just not logical to me and wanted to report it. Thanks for your help.

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1419) Trigger Rule not respected downstream of BranchPythonOperator

2018-11-15 Thread Conrad Lee (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688014#comment-16688014
 ] 

Conrad Lee commented on AIRFLOW-1419:
-

[~xnuinside] thanks for having a look.  Also thanks for fining the bug in the 
example code–I've fixed that.

 

I'm not sure this should be closed though.  As I recall, before 1.8.2, no dummy 
operator was required at all, because task-skip propagated differently.  When 
1.8.2 came along, all of a sudden a dummy was necessary – the question is 
whether this is desired.

I much preferred this previous behavior–why should a dummy operator be 
necessary at all?  If one of the child tasks has a trigger rule thats stops the 
propagation of task-skipping (such as ALL_DONE), then IMHO it should never be 
skipped.

> Trigger Rule not respected downstream of BranchPythonOperator
> -
>
> Key: AIRFLOW-1419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1419
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Conrad Lee
>Priority: Major
>
> Lets consider the following DAG:
> {noformat}
>   
>  /  \
> branch_op confluence_op
>  \__work_op/
> {noformat}
> This is implemented in the following code:
> {code:java}
> import airflow
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.utils.trigger_rule import TriggerRule
> from airflow.models import DAG
> args = {
> 'owner': 'airflow',
> 'start_date': airflow.utils.dates.days_ago(2)
> }
> dag = DAG(
> dag_id='branch_skip_problem',
> default_args=args,
> schedule_interval="@daily")
> branch_op = BranchPythonOperator(
> task_id='branch_op',
> python_callable=lambda: 'work_op',
> dag=dag)
> work_op = DummyOperator(task_id='work_op', dag=dag)
> confluence_op = DummyOperator(task_id='confluence_op', dag=dag, 
> trigger_rule=TriggerRule.ALL_DONE)
> branch_op.set_downstream(confluence_op)
> branch_op.set_downstream(work_op)
> work_op.set_downstream(confluence_op)
> {code}
> Note that branch_op is a BranchPythonOperator, work_op and confluence_op are 
> DummyOperators, and that confluence_op has its trigger_rule set to ALL_DONE.
> In dag runs where brancher_op chooses to activate work_op as its child, 
> confluence_op never runs. This doesn't seem right, because confluence_op has 
> two parents and a trigger_rule set that it'll run as soon as all of its 
> parents are done (whether or not they are skipped).
> I know this example seems contrived and that in practice there are better 
> ways of conditionally executing work_op. However, this is the minimal code to 
> illustrate the problem. You can imagine that this problem might actually 
> creep up in practice where originally there was a good reason to use the 
> BranchPythonOperator, and then time passes and someone modifies one of the 
> branches so that it doesn't really contain any children anymore, thus 
> resembling the example.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688009#comment-16688009
 ] 

Greg H commented on AIRFLOW-3182:
-

Thanks for your input. I'm new to Airflow and I agree that I may not fully 
understand how some items work (and yes, I did read the docs). Of course, this 
case may be refactored and done different ways.
 
What I understand, the BranhcOperator here is wrong because we're not using 
conditional statement, right? OK, having now the explicit condition, that btw. 
does the same thing as previously, results in the same behaviour...

{code}
def branch():
for x in range(0, 24):
if x == get_current_hour():
return task_prefix + str(x)
{code}

It's just strange to me that some downstream task is run despite it's not 
connected to the executed upstream one ('daily_aggregation' and 
'task_four_hour-7' as in attached screen shot). If you think that this 
behaviour is correct or my example is still wrong, then please reject this 
Jira. 

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor

2018-11-15 Thread Stefan Seelmann (JIRA)

Stefan Seelmann created AIRFLOW-3353:


 Summary: redis-py 3.0.0 dependency breaks celery executor
 Key: AIRFLOW-3353
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
 Project: Apache Airflow
  Issue Type: Bug
  Components: celery
Affects Versions: 1.10.0
Reporter: Stefan Seelmann


redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so 
installs redis-py 3.0.0 now.

Error in worker below.

Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)

{code}
[2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error: 
AttributeError("'float' object has no attribute 'items'",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 
205, in start
self.blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, 
in start
step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, 
in start
return self.obj.start()
  File 
"/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
line 317, in start
blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, 
in start
step.start(parent)
  File 
"/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", 
line 593, in start
c.loop(*c.loop_args())
  File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 
91, in asynloop
next(loop)
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 
354, in create_loop
cb(*cbargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 
1040, in on_readable
self.cycle.on_readable(fileno)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 
337, in on_readable
chan.handlers[type]()
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 
724, in _brpop_read
self.connection._deliver(loads(bytes_to_str(item)), dest)
  File 
"/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 
983, in _deliver
callback(message)
  File 
"/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 
632, in _callback
self.qos.append(message, message.delivery_tag)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 
149, in append
pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, in 
zadd
for pair in iteritems(mapping):
  File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, in 
iteritems
return iter(x.items())
AttributeError: 'float' object has no attribute 'items'

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks

2018-11-15 Thread Iuliia Volkova (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687993#comment-16687993
 ] 

Iuliia Volkova commented on AIRFLOW-3182:
-

[~Zeckt], I got your case, you need after your branch one more branch task - 
where you need to check if it already 23 hour or not. This BranchTask needs to 
be upstream on all hours. And this task will downstream Your aggregation task. 

> 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream 
> tasks
> --
>
> Key: AIRFLOW-3182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3182
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Greg H
>Priority: Major
> Attachments: BrannchPythonOperator.png, Screen Shot 2018-11-15 at 
> 13.51.07.png
>
>
> We have a job that runs some data processing every hour. At the end of the 
> day we need to run aggregation on all data generated by the 'hourly' jobs, 
> regardless if any 'hourly' job failed or not. For this purpose we have 
> prepared DAG that uses BranchPythonOperator in order to decide which 'hourly' 
> job needs to be run in given time and when task for hour 23 is done, we 
> trigger the aggregation (downstream). For this to work regardless of the last 
> 'hourly' task status the *'all_done'* trigger rule is set in the aggregation 
> task. Unfortunately, such configuration works incorrectly causing aggregation 
> task to be run after every 'hourly' task, despite the fact the aggregation 
> task is set as downstream for 'task_for_hour-23' +only+:
>   !BrannchPythonOperator.png!
> Here's sample code:
> {code:java}
> # coding: utf-8
> from airflow import DAG
> from airflow.operators.python_operator import PythonOperator
> from airflow.operators.python_operator import BranchPythonOperator
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.models import TriggerRule
> from datetime import datetime
> import logging
> dag_id = 'test'
> today = datetime.today().strftime("%Y-%m-%d");
> task_prefix = 'task_for_hour-'
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2018, 6, 18),
> 'catchup': False,
> }
> dag = DAG(
> dag_id=dag_id,
> default_args=default_args,
> schedule_interval="@hourly",
> catchup=False
> )
> # Setting the current hour
> def get_current_hour():
> return datetime.now().hour
> # Returns the name id of the task to launch next (task_for_hour-0, 
> task_for_hour-1, etc.)
> def branch():
> return task_prefix + str(get_current_hour())
> # Running hourly job
> def run_hourly_job(**kwargs):
> current_hour = get_current_hour()
> logging.info("Running job for hour: %s" % current_hour)
> # Main daily aggregation
> def run_daily_aggregation(**kwargs):
> logging.info("Running daily aggregation for %s" % today)
> 
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> # 'branch' method returns name of the task to be run next.
> hour_branching = BranchPythonOperator(
> task_id='hour_branching',
> python_callable=branch,
> dag=dag)
> run_aggregation = PythonOperator(
> task_id='daily_aggregation',
> python_callable=run_daily_aggregation,
> provide_context=True,
> trigger_rule=TriggerRule.ALL_DONE,
> dag=dag
> )
> start_task.set_downstream(hour_branching)
> # Create tasks for each hour
> for hour in range(24):
> if hour == 23:
> task_for_hour_23 = PythonOperator(
> task_id=task_prefix + '23',
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag
> )
> hour_branching.set_downstream(task_for_hour_23)
> task_for_hour_23.set_downstream(run_aggregation)
> else:
> hour_branching.set_downstream(PythonOperator(
> task_id=task_prefix + str(hour),
> python_callable=run_hourly_job,
> provide_context=True,
> dag=dag)
> )
> {code}
> This me be also related to AIRFLOW-1419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] verdan commented on issue #4142: [AIRFLOW-3303] Deprecate old UI in favor of FAB

verdan commented on issue #4142: [AIRFLOW-3303] Deprecate old UI in favor of FAB
URL: 
https://github.com/apache/incubator-airflow/pull/4142#issuecomment-439029924
 
 
   @ron819 can you please double check if what you are saying is the correct 
reference of the failing tests ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3182) 'all_done' trigger rule works incorrectly with BranchPythonOperator upstream tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg H updated AIRFLOW-3182:

Description: 
We have a job that runs some data processing every hour. At the end of the day 
we need to run aggregation on all data generated by the 'hourly' jobs, 
regardless if any 'hourly' job failed or not. For this purpose we have prepared 
DAG that uses BranchPythonOperator in order to decide which 'hourly' job needs 
to be run in given time and when task for hour 23 is done, we trigger the 
aggregation (downstream). For this to work regardless of the last 'hourly' task 
status the *'all_done'* trigger rule is set in the aggregation task. 
Unfortunately, such configuration works incorrectly causing aggregation task to 
be run after every 'hourly' task, despite the fact the aggregation task is set 
as downstream for 'task_for_hour-23' +only+:

  !BrannchPythonOperator.png!

Here's sample code:
{code:java}
# coding: utf-8
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.models import TriggerRule
from datetime import datetime
import logging

dag_id = 'test'

today = datetime.today().strftime("%Y-%m-%d");
task_prefix = 'task_for_hour-'

default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2018, 6, 18),
'catchup': False,
}

dag = DAG(
dag_id=dag_id,
default_args=default_args,
schedule_interval="@hourly",
catchup=False
)

# Setting the current hour
def get_current_hour():
return datetime.now().hour


# Returns the name id of the task to launch next (task_for_hour-0, 
task_for_hour-1, etc.)
def branch():
return task_prefix + str(get_current_hour())


# Running hourly job
def run_hourly_job(**kwargs):
current_hour = get_current_hour()
logging.info("Running job for hour: %s" % current_hour)


# Main daily aggregation
def run_daily_aggregation(**kwargs):
logging.info("Running daily aggregation for %s" % today)


start_task = DummyOperator(
task_id='start',
dag=dag
)

# 'branch' method returns name of the task to be run next.
hour_branching = BranchPythonOperator(
task_id='hour_branching',
python_callable=branch,
dag=dag)

run_aggregation = PythonOperator(
task_id='daily_aggregation',
python_callable=run_daily_aggregation,
provide_context=True,
trigger_rule=TriggerRule.ALL_DONE,
dag=dag
)

start_task.set_downstream(hour_branching)

# Create tasks for each hour
for hour in range(24):
if hour == 23:
task_for_hour_23 = PythonOperator(
task_id=task_prefix + '23',
python_callable=run_hourly_job,
provide_context=True,
dag=dag
)

hour_branching.set_downstream(task_for_hour_23)
task_for_hour_23.set_downstream(run_aggregation)
else:
hour_branching.set_downstream(PythonOperator(
task_id=task_prefix + str(hour),
python_callable=run_hourly_job,
provide_context=True,
dag=dag)
)
{code}
This me be also related to AIRFLOW-1419

  was:
We have a job that runs some data processing every hour. At the end of the day 
we need to run aggregation on all data generated by the 'hourly' jobs, 
regardless if any 'hourly' job failed or not. For this purpose we have prepared 
DAG that uses BranchPythonOperator in order to decide which 'hourly' job needs 
to be run in given time and when task for hour 23 is done, we trigger the 
aggregation (downstream). For this to work regardless of the last 'hourly' task 
status the *'all_done'* trigger rule is set in the aggregation task. 
Unfortunately, such configuration works incorrectly causing aggregation task to 
be run after every 'hourly' task, despite the fact the aggregation task is set 
as downstream for 'task_for_hour-23' +only+:

  !BrannchPythonOperator.png!

Here's sample code:
{code:java}
# coding: utf-8
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.models import TriggerRule
from datetime import datetime
import logging

dag_id = 'test'

today = datetime.today().strftime("%Y-%m-%d");
task_prefix = 'task_for_hour-'

default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2018, 6, 18),
'catchup': False,
}

dag = DAG(
dag_id=dag_id,
default_args=default_args,
schedule_interval="@hourly",
catchup=False
)

# Setting the current hour
def get_current_hour():
return datetime.now().hour


# Returns the name id of the task to launch next (task_for_hour-0, 
task_for_hour-1, etc.)
def branch():
return task_prefix +

[GitHub] phani8996 commented on issue #4193: Fix showing config on RBAC UI when expose_config is False

phani8996 commented on issue #4193: Fix showing config on RBAC UI when 
expose_config is False
URL: 
https://github.com/apache/incubator-airflow/pull/4193#issuecomment-439028830
 
 
   Closed PR as i pushed code to wrong branch


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 opened a new pull request #4194: [AIRFLOW-3522] Fix showing config on RBAC UI when expose_config is False

phani8996 opened a new pull request #4194: [AIRFLOW-3522] Fix showing config on
RBAC UI when expose_config is False
URL: https://github.com/apache/incubator-airflow/pull/4194

RBAC UI does not respect variable `expose_config` in airflow config to show
configuration.
This snippet will show config only when `expose_config` is set to `True`.

Make sure you have checked _all_ steps below.

### Jira

- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3352\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3352

### Description

- [x] Here are some details about my PR, including screenshots of any UI
- This PR will control showing airflow configuration RBAC UI based on
variable `expose_config` defined in airflow config
- Please find attached screenshot for the same

![img-20181115-wa](https://user-images.githubusercontent.com/12140904/48553561-f0dc2b00-e901-11e8-93b5-db2579e7434f.jpg)

### Tests

- [x] My PR does not need testing as this is UI change.

### Commits

- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Code Quality

- [x] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] phani8996 closed pull request #4193: Fix showing config on RBAC UI when expose_config is False

phani8996 closed pull request #4193: Fix showing config on RBAC UI when 
expose_config is False
URL: https://github.com/apache/incubator-airflow/pull/4193
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 29e8da1b9c..f5ca0ca0c7 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -1798,11 +1798,18 @@ def conf(self):
 raw = request.args.get('raw') == "true"
 title = "Airflow Configuration"
 subtitle = conf.AIRFLOW_CONFIG
-with open(conf.AIRFLOW_CONFIG, 'r') as f:
-config = f.read()
-table = [(section, key, value, source)
- for section, parameters in conf.as_dict(True, True).items()
- for key, (value, source) in parameters.items()]
+# Don't show config when expose_config variable is False in airflow 
config
+if conf.getboolean("webserver", "expose_config"):
+with open(conf.AIRFLOW_CONFIG, 'r') as f:
+config = f.read()
+table = [(section, key, value, source)
+ for section, parameters in conf.as_dict(True, 
True).items()
+ for key, (value, source) in parameters.items()]
+else:
+config = (
+"# Your Airflow administrator chose not to expose the "
+"configuration, most likely for security reasons.")
+table = None
 
 if raw:
 return Response(


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 opened a new pull request #4193: Fix showing config on RBAC UI when expose_config is False