[jira] [Commented] (AIRFLOW-3419) S3_hook.select_key is broken on Python3

[jack (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748460#comment-16748460
 ] 

jack commented on AIRFLOW-3419:
---

[~sekikn] guess you have a go :)

> S3_hook.select_key is broken on Python3
> ---
>
> Key: AIRFLOW-3419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3419
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: boto3, hooks
>Affects Versions: 1.10.1
>Reporter: Maria Rebelka
>Priority: Major
>
> Hello,
> Using select_key throws an error:
> {quote}text = S3Hook('aws_conn').select_key(key='my_key',
>                                      bucket_name='my_bucket',
>                                      expression='SELECT * FROM S3Object s',
>                                      expression_type='SQL',
>                                      input_serialization={'JSON': \{'Type': 
> 'DOCUMENT'}},
>                                      output_serialization={'JSON': {}}){quote}
> Traceback (most recent call last):
> {quote}   File "db.py", line 31, in 
> output_serialization={'JSON': {}})
>   File "/usr/local/lib/python3.4/site-packages/airflow/hooks/S3_hook.py", 
> line 262, in select_key
> for event in response['Payload']
> TypeError: sequence item 0: expected str instance, bytes found{quote}
> Seems that the problem is in this line:
> S3_hook.py, line 262:  return ''.join(event['Records']['Payload']
> which probably should be return 
> ''.join(event['Records']['Payload'].decode('utf-8')
> From example in Amazon blog:
> https://aws.amazon.com/blogs/aws/s3-glacier-select/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456291898
 
 
   @XD-DENG , I think it shouldn't matter with npm. We only use npm to build 
the static access which may just cause UI off. But I try with v-10-test branch 
which I don't see any issues.
   
   @kaxil , do you think you could try out this branch on top v-10-test and see 
if you still see the issues or not?
   
   cc @seelmann  and see if you could try out and see whether the viewer issue 
still exists or not.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init variable i avoid raise NameError

[GitBox]

zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init 
variable i avoid raise NameError
URL: https://github.com/apache/airflow/pull/4560#discussion_r249659195
 
 

 ##
 File path: airflow/hooks/hive_hooks.py
 ##
 @@ -878,6 +878,7 @@ def to_csv(
 header = next(results_iter)
 message = None
 
+i = 0
 
 Review comment:
   @feng-tao Of course we can write some code like
   ```python
   self.log.info("Done. Loaded a total of %s rows.", i if 'i' in locals() else 
0)
   ```
   but I think is a little weird, so I put `i = 0` in L881


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init variable i avoid raise NameError

[GitBox]

zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init 
variable i avoid raise NameError
URL: https://github.com/apache/airflow/pull/4560#discussion_r249659195
 
 

 ##
 File path: airflow/hooks/hive_hooks.py
 ##
 @@ -878,6 +878,7 @@ def to_csv(
 header = next(results_iter)
 message = None
 
+i = 0
 
 Review comment:
   @feng-tao Of course we can write some code like
   ```python
   self.log.info("Done. Loaded a total of %s rows.", i if 'i' in locals() else 
0)
   ```
   but I think is a little strange, so I put `i = 0` in L881


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init variable i avoid raise NameError

[GitBox]

zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init 
variable i avoid raise NameError
URL: https://github.com/apache/airflow/pull/4560#discussion_r249658437
 
 

 ##
 File path: airflow/hooks/hive_hooks.py
 ##
 @@ -878,6 +878,7 @@ def to_csv(
 header = next(results_iter)
 message = None
 
+i = 0
 
 Review comment:
   @feng-tao We could not change L904 due to we don't know if `i` exists or 
not. set `i = 0` just init `i`.
   If hql statement return no reocrd, L892 will not run and `i = 0`
   else hql statement have record and `i` is reocrd number.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init variable i avoid raise NameError

[GitBox]

zhongjiajie commented on a change in pull request #4560: [AIRFLOW-3733] Init 
variable i avoid raise NameError
URL: https://github.com/apache/airflow/pull/4560#discussion_r249658437
 
 

 ##
 File path: airflow/hooks/hive_hooks.py
 ##
 @@ -878,6 +878,7 @@ def to_csv(
 header = next(results_iter)
 message = None
 
+i = 0
 
 Review comment:
   We could not change L904 due to we don't know if `i` exists or not. set `i = 
0` just init `i`.
   If hql statement return no reocrd, L892 will not run and `i = 0`
   else hql statement have record and `i` is reocrd number.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456285488
 
 
   @feng-tao yes, that's what I applied. But please note what I did was 
applying it against the `www_rbac/security.py` in `1.10.2rc3` (on biz trip with 
a laptop that I can't install npm). This may be why we got different results?
   
   I trust your testing. May you please test this fix on `1-10-test` as well in 
case any conflict? Thank you.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ffinfo commented on a change in pull request #4562: [WIP][AIRFLOW-3735] Separated parsing from checking

[GitBox]

ffinfo commented on a change in pull request #4562: [WIP][AIRFLOW-3735] 
Separated parsing from checking
URL: https://github.com/apache/airflow/pull/4562#discussion_r249649280
 
 

 ##
 File path: airflow/utils/dag_processing.py
 ##
 @@ -43,16 +46,18 @@
 
 # To avoid circular imports
 import airflow.models
-from airflow import configuration as conf
+from airflow import configuration as conf, configuration
 from airflow.dag.base_dag import BaseDag, BaseDagBag
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, AirflowDagCycleException
 from airflow.models import errors
 from airflow.settings import logging_class_path
 from airflow.utils import timezone
 from airflow.utils.db import provide_session
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow.utils.state import State
+from airflow.utils.timeout import timeout
 
+_log = logging.root.getChild(__package__)
 
 Review comment:
   thank you


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #4560: [AIRFLOW-3733] Init variable i avoid raise NameError

[GitBox]

feng-tao commented on a change in pull request #4560: [AIRFLOW-3733] Init 
variable i avoid raise NameError
URL: https://github.com/apache/airflow/pull/4560#discussion_r249647958
 
 

 ##
 File path: airflow/hooks/hive_hooks.py
 ##
 @@ -878,6 +878,7 @@ def to_csv(
 header = next(results_iter)
 message = None
 
+i = 0
 
 Review comment:
   if log error, shouldn't we change L904 instead? If we set i = 0, then it 
will get shadowed in L892.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456277442
 
 
   @XD-DENG , could you confirm if you apply this 
commit(https://github.com/apache/airflow/commit/31bfb65db3282ca31253d38228d0fe4a6de1dcef)?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456277058
 
 
   This is the UI for a viewer / user (which doesn't have the security tab) on 
my end. Do you apply the latest one?
   https://user-images.githubusercontent.com/3223098/51514428-8f827d80-1dc4-11e9-90a0-ae9940c102a3.png;>
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456277058
 
 
   # This is the UI for a viewer / user (which doesn't have the security tab). 
Do you apply the latest one?
   https://user-images.githubusercontent.com/3223098/51514428-8f827d80-1dc4-11e9-90a0-ae9940c102a3.png;>
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456276192
 
 
   Hi @feng-tao , I tried to apply your latest change. But the issue seems 
still there. Using `Viewer`/`User`, I can still see `Security`/`Admin` menus, 
and edit inside.
   
   I'm using SQLite. Before testing, I have deleted the earlier .db file and 
run `initdb` again.
   
   Not sure if I missed anything. @kaxil please help test as well so we can 
confirm.
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

codecov-io edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able 
to view dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456257845
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=h1) 
Report
   > Merging 
[#4569](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/98e852219fc73c7ec049feeab7305bc7c0e89698?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `94.11%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4569/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4569  +/-   ##
   ==
   + Coverage   74.12%   74.13%   +0.01% 
   ==
 Files 421  421  
 Lines   2770027715  +15 
   ==
   + Hits2053320547  +14 
   - Misses   7167 7168   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4569/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `92.89% <94.11%> (+0.04%)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4569/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.33% <0%> (-0.05%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/4569/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `59.92% <0%> (+0.17%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=footer). 
Last update 
[98e8522...31bfb65](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456269760
 
 
   CI pass(k8s one is random, transient)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2697) Drop snakebite in favour of hdfs3

[zhongjiajie (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748370#comment-16748370
 ] 

zhongjiajie commented on AIRFLOW-2697:
--

Awesome I would like to see this moving forward.

> Drop snakebite in favour of hdfs3
> -
>
> Key: AIRFLOW-2697
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2697
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Julian de Ruiter
>Assignee: Julian de Ruiter
>Priority: Major
>
> The current HdfsHook relies on the snakebite library, which is unfortunately 
> not compatible with Python 3. To add Python 3 support for the HdfsHook 
> requires switching to a different library for interacting with HDFS. The 
> hdfs3 library is an attractive alternative, as it supports Python 3 and seems 
> to be stable and relatively well supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] zhongjiajie edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

[GitBox]

zhongjiajie edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in 
favour of hdfs3
URL: https://github.com/apache/airflow/pull/3560#issuecomment-456264096
 
 
   This PR is Great, I would love to see this moving forward


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhongjiajie commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

[GitBox]

zhongjiajie commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour 
of hdfs3
URL: https://github.com/apache/airflow/pull/3560#issuecomment-456264096
 
 
   This PR is Great, I would love to have this moving forward


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

codecov-io commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456257845
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@98e8522`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `93.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4569/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#4569   +/-   ##
   =
 Coverage  ?   74.13%   
   =
 Files ?  421   
 Lines ?27714   
 Branches  ?0   
   =
 Hits  ?20546   
 Misses? 7168   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4569/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `92.85% <93.33%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=footer). 
Last update 
[98e8522...27ff881](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

codecov-io commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456257842
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@98e8522`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `93.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4569/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#4569   +/-   ##
   =
 Coverage  ?   74.13%   
   =
 Files ?  421   
 Lines ?27714   
 Branches  ?0   
   =
 Hits  ?20546   
 Misses? 7168   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4569/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `92.85% <93.33%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=footer). 
Last update 
[98e8522...27ff881](https://codecov.io/gh/apache/airflow/pull/4569?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao merged pull request #4570: [AIRFLOW-XXX] Fix typos

[GitBox]

feng-tao merged pull request #4570: [AIRFLOW-XXX] Fix typos
URL: https://github.com/apache/airflow/pull/4570
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456256456
 
 
   PTAL @XD-DENG , @kaxil , code updated with unit test. I have tried on my end 
which works(viewer couldn't see the Admin role menu). Could you test it on your 
end and see if there are any issues?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] shubhparekh commented on issue #4439: [AIRFLOW-3626] Fixed zipped Dag trigger

[GitBox]

shubhparekh commented on issue #4439: [AIRFLOW-3626] Fixed zipped Dag trigger
URL: https://github.com/apache/airflow/pull/4439#issuecomment-456253662
 
 
   > This needs tests please so it doesn't break again. There is a 
`tests/dags/test_zip.zip` already in the repo that contains a dag or two that 
you can use in the tests.
   
   I will have a look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dongli commented on issue #4507: [AIRFLOW-3391] Upgrade Pendulum to latest version

[GitBox]

dongli commented on issue #4507: [AIRFLOW-3391] Upgrade Pendulum to latest 
version
URL: https://github.com/apache/airflow/pull/4507#issuecomment-456248209
 
 
   Any news?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456241608
 
 
   Thanks @feng-tao 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456241430
 
 
   @XD-DENG  correct, I don't think that is the correct behavior. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456241343
 
 
   thanks @XD-DENG , @kaxil , I know what is the issue now. I will work on a pr 
with test tonight.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456240412
 
 
   Hi @feng-tao , I tried to cherry-pick this change into 1.10.2rc3 (I copied 
the change in your PR in `airflow/www_rbac/security.py` of 1.10.2rc3), seems 
something is wrong:
   
   **I tried to log in using accounts with `User`/`Viewer` role, I can see 
`Security`/`Admin` tabs, and can even create new user or change permission of 
each roles.** 
   
   I believe this is not desired. Please try and let me know if you can 
reproduce this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456240412
 
 
   Hi @feng-tao , I tried to cherry-pick this change into 1.10.2rc3 (I copied 
the change in your PR in `airflow/www_rbac/security.py` of 1.10.2rc3), seems 
something is wrong:
   **I tried to login using accounts with `User`/`Viewer` role, I can see 
security tab, and can even create new user or change permission of each 
roles.** 
   
   I believe this is not desired. Please try and let me know if you can 
reproduce this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456240412
 
 
   Hi @feng-tao , I tried to cherry-pick this change into 1.10.2rc3 (I copied 
the change in your PR in `airflow/www_rbac/security.py` of 1.10.2rc3), seems 
something is wrong:
   **I tried to log in using accounts with `User`/`Viewer` role, I can see 
security tab, and can even create new user or change permission of each 
roles.** 
   
   I believe this is not desired. Please try and let me know if you can 
reproduce this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-456240412
 
 
   Hi @feng-tao , I tried to cherry-pick this change into 1.10.2rc3 (I made the 
change in your PR in `airflow/www_rbac/security.py` of 1.10.2rc3), seems 
something is wrong:
   **I tried to login using accounts with `User`/`Viewer` role, I can see 
security tab, and can even create new user or change permission of each 
roles.** 
   
   I believe this is not desired. Please try and let me know if you can 
reproduce this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure Hdinsight operators for creating cluster and submitting

[GitBox]

mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure 
Hdinsight operators for creating cluster and submitting
URL: https://github.com/apache/airflow/pull/4460#discussion_r249614469
 
 

 ##
 File path: airflow/contrib/hooks/azure_hdinsight_hook.py
 ##
 @@ -0,0 +1,252 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import json
+import logging
+import time
+
+import requests
+import urllib3
+from azure.common.credentials import ServicePrincipalCredentials
+from azure.mgmt.hdinsight import HDInsightManagementClient
+from azure.mgmt.resource import ResourceManagementClient
+from azure.mgmt.resource.resources.models import DeploymentMode, 
DeploymentProperties
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+
+class AzureHDInsightHook(BaseHook, LoggingMixin):
+
+def __init__(self, azure_conn_id='azure_hdinsight_default'):
 
 Review comment:
   Good practice for GCP are available: 
https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit
   However, the document is still developing all the time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow

[GitBox]

mik-laj commented on a change in pull request #4068: [AIRFLOW-2310]: Add AWS 
Glue Job Compatibility to Airflow
URL: https://github.com/apache/airflow/pull/4068#discussion_r249612395
 
 

 ##
 File path: airflow/contrib/hooks/aws_glue_job_hook.py
 ##
 @@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+import time
+from botocore.exceptions import ClientError
+
+
+class AwsGlueJobHook(AwsHook):
+"""
+Interact with AWS Glue - create job
+
+:param job_name: unique job name per AWS account
+:type str
+:param desc: job description
+:type str
+:param aws_conn_id: aws connection id
+:type aws_conn_id: str
+"""
+
+def __init__(self,
+ job_name=None,
+ desc=None,
+ aws_conn_id='aws_default',
+ *args, **kwargs):
+self.job_name = job_name
+self.desc = desc
+self.aws_conn_id = aws_conn_id
+super(AwsGlueJobHook, self).__init__(self.aws_conn_id, *args, **kwargs)
+
+def get_conn(self):
+return self.get_client_type('glue')
+
+def list_jobs(self):
+conn = self.get_conn()
+return conn.get_jobs()
+
+def initialize_job(self, script_arguments=None):
+"""
+Initializes connection with AWS Glue
+to run job
+:return:
+"""
+glue_client = self.get_conn()
+
+try:
+job_response = self.get_glue_job()
+job_name = job_response['Name']
+job_run = glue_client.start_job_run(
+JobName=job_name,
+Arguments=script_arguments
+)
+return self.job_completion(job_name, job_run['JobRunId'])
+except ClientError as general_error:
+raise AirflowException(
+'Failed to run aws glue job, error: {error}'.format(
+error=str(general_error)
+)
+)
+
+def job_completion(self, job_name=None, run_id=None):
 
 Review comment:
   Could you correct the name of this method to include a verb?  
`GoogleCloudBaseHook` use a name 
   `_wait_for_operation_to_complete`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure Hdinsight operators for creating cluster and submitting

[GitBox]

mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure 
Hdinsight operators for creating cluster and submitting
URL: https://github.com/apache/airflow/pull/4460#discussion_r249609606
 
 

 ##
 File path: airflow/contrib/hooks/azure_hdinsight_hook.py
 ##
 @@ -0,0 +1,252 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import json
+import logging
+import time
+
+import requests
+import urllib3
+from azure.common.credentials import ServicePrincipalCredentials
+from azure.mgmt.hdinsight import HDInsightManagementClient
+from azure.mgmt.resource import ResourceManagementClient
+from azure.mgmt.resource.resources.models import DeploymentMode, 
DeploymentProperties
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+
+class AzureHDInsightHook(BaseHook, LoggingMixin):
+
+def __init__(self, azure_conn_id='azure_hdinsight_default'):
 
 Review comment:
   Otherwise. Integration into the Azure platform is at an early stage of 
development. There are no good practices yet. I ask a lot of questions to force 
a discussion that can create these practices.
   Could you review other unacceptable patches? I am happy to hear your opinion.
   https://github.com/apache/airflow/pulls?q=is%3Apr+Azure+is%3Aopen


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure Hdinsight operators for creating cluster and submitting

[GitBox]

mik-laj commented on a change in pull request #4460: [AIRFLOW-3604] Add Azure 
Hdinsight operators for creating cluster and submitting
URL: https://github.com/apache/airflow/pull/4460#discussion_r249608574
 
 

 ##
 File path: airflow/contrib/hooks/azure_hdinsight_hook.py
 ##
 @@ -0,0 +1,252 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import json
+import logging
+import time
+
+import requests
+import urllib3
+from azure.common.credentials import ServicePrincipalCredentials
+from azure.mgmt.hdinsight import HDInsightManagementClient
+from azure.mgmt.resource import ResourceManagementClient
+from azure.mgmt.resource.resources.models import DeploymentMode, 
DeploymentProperties
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+
+class AzureHDInsightHook(BaseHook, LoggingMixin):
+
+def __init__(self, azure_conn_id='azure_hdinsight_default'):
 
 Review comment:
   I do not know it well, so I am not answering your question. But maybe you 
should separate these two things if one is not very dependent on the other? I 
mean creating a separate HDP hook. Can you imagine that someone will use this 
operator without the Azure platform?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4562: [WIP][AIRFLOW-3735] Separated parsing from checking

[GitBox]

mik-laj commented on a change in pull request #4562: [WIP][AIRFLOW-3735] 
Separated parsing from checking
URL: https://github.com/apache/airflow/pull/4562#discussion_r249605814
 
 

 ##
 File path: airflow/utils/dag_processing.py
 ##
 @@ -43,16 +46,18 @@
 
 # To avoid circular imports
 import airflow.models
-from airflow import configuration as conf
+from airflow import configuration as conf, configuration
 from airflow.dag.base_dag import BaseDag, BaseDagBag
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, AirflowDagCycleException
 from airflow.models import errors
 from airflow.settings import logging_class_path
 from airflow.utils import timezone
 from airflow.utils.db import provide_session
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow.utils.state import State
+from airflow.utils.timeout import timeout
 
+_log = logging.root.getChild(__package__)
 
 Review comment:
   Look at: 
   
https://github.com/apache/airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/airflow/api/__init__.py#L30


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

codecov-io commented on issue #4514: [AIRFLOW-3698] Add documentation for AWS 
Connection
URL: https://github.com/apache/airflow/pull/4514#issuecomment-456216608
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=h1) 
Report
   > Merging 
[#4514](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0949cbd4d9e57abf9d4c2f439d34b9703cb9a05?src=pr=desc)
 will **increase** coverage by `0.31%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4514/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4514  +/-   ##
   ==
   + Coverage   73.81%   74.12%   +0.31% 
   ==
 Files 421  421  
 Lines   2766227700  +38 
   ==
   + Hits2041820532 +114 
   + Misses   7244 7168  -76
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `64.64% <0%> (-0.44%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5)
 | `75.75% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `79.82% <0%> (+0.08%)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.33% <0%> (+0.14%)` | :arrow_up: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `92.3% <0%> (+0.2%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `79.03% <0%> (+0.8%)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `75.26% <0%> (+1.84%)` | :arrow_up: |
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `64.02% <0%> (+1.99%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `81.39% <0%> (+3.78%)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `81.81% <0%> (+4.54%)` | :arrow_up: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=footer). 
Last update 
[c0949cb...742914d](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

codecov-io commented on issue #4514: [AIRFLOW-3698] Add documentation for AWS 
Connection
URL: https://github.com/apache/airflow/pull/4514#issuecomment-456216603
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=h1) 
Report
   > Merging 
[#4514](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0949cbd4d9e57abf9d4c2f439d34b9703cb9a05?src=pr=desc)
 will **increase** coverage by `0.31%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4514/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4514  +/-   ##
   ==
   + Coverage   73.81%   74.12%   +0.31% 
   ==
 Files 421  421  
 Lines   2766227700  +38 
   ==
   + Hits2041820532 +114 
   + Misses   7244 7168  -76
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `64.64% <0%> (-0.44%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5)
 | `75.75% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `79.82% <0%> (+0.08%)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.33% <0%> (+0.14%)` | :arrow_up: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `92.3% <0%> (+0.2%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `79.03% <0%> (+0.8%)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `75.26% <0%> (+1.84%)` | :arrow_up: |
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `64.02% <0%> (+1.99%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `81.39% <0%> (+3.78%)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `81.81% <0%> (+4.54%)` | :arrow_up: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/4514/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=footer). 
Last update 
[c0949cb...742914d](https://codecov.io/gh/apache/airflow/pull/4514?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4570: [AIRFLOW-XXX] Fix typos

[GitBox]

codecov-io commented on issue #4570: [AIRFLOW-XXX] Fix typos
URL: https://github.com/apache/airflow/pull/4570#issuecomment-456211807
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=h1) 
Report
   > Merging 
[#4570](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/5084305e31072fc06ae3415e7e5f4be755e1d5a7?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4570/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4570  +/-   ##
   ==
   - Coverage   74.12%   74.12%   -0.01% 
   ==
 Files 421  421  
 Lines   2770027700  
   ==
   - Hits2053420532   -2 
   - Misses   7166 7168   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4570/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=)
 | `83.75% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4570/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.33% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=footer). 
Last update 
[5084305...18af05f](https://codecov.io/gh/apache/airflow/pull/4570?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] danabananarama commented on issue #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

danabananarama commented on issue #4514: [AIRFLOW-3698] Add documentation for 
AWS Connection
URL: https://github.com/apache/airflow/pull/4514#issuecomment-456208799
 
 
   Thanks for the feedback @ashb, @XD-DENG , I think I've addressed all the 
comments. Please let me know if anything else needs tweaking!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] danabananarama commented on a change in pull request #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

danabananarama commented on a change in pull request #4514: [AIRFLOW-3698] Add 
documentation for AWS Connection
URL: https://github.com/apache/airflow/pull/4514#discussion_r249586986
 
 

 ##
 File path: docs/howto/manage-connections.rst
 ##
 @@ -151,6 +151,52 @@ Scopes (comma separated)
 issue `AIRFLOW-2522
 `_.
 
+Amazon Web Services
+~~~
+
+The Amazon Web Services connection type enables the :ref:`AWS Integrations
+`.
+
+Authenticating to AWS
+'
+
+Authentication may be performed using any of the `boto3 options 
`_.
 Alternatively, one can pass credentials in as a Connection initialisation 
parameter.
+
+Default Connection IDs
+''
+
+The default connection ID is ``aws_default``.
+
+Configuring the Connection
+''
+
+Login (optional)
+Specify the AWS access key ID.
+
+Password (optional)
+Specify the AWS secret access key.
+
+Extra (optional)
 
 Review comment:
   ah I missed that, thanks for the spot!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] danabananarama commented on a change in pull request #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

danabananarama commented on a change in pull request #4514: [AIRFLOW-3698] Add 
documentation for AWS Connection
URL: https://github.com/apache/airflow/pull/4514#discussion_r249586640
 
 

 ##
 File path: docs/howto/manage-connections.rst
 ##
 @@ -151,6 +151,52 @@ Scopes (comma separated)
 issue `AIRFLOW-2522
 `_.
 
+Amazon Web Services
+~~~
+
+The Amazon Web Services connection type enables the :ref:`AWS Integrations
+`.
+
+Authenticating to AWS
+'
+
+Authentication may be performed using any of the `boto3 options 
`_.
 Alternatively, one can pass credentials in as a Connection initialisation 
parameter.
+
+Default Connection IDs
+''
+
+The default connection ID is ``aws_default``.
+
+Configuring the Connection
+''
+
+Login (optional)
+Specify the AWS access key ID.
+
+Password (optional)
+Specify the AWS secret access key.
+
+Extra (optional)
+Specify the extra parameters (as json dictionary) that can be used in ssh
 
 Review comment:
   d'oh ><


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ffinfo commented on a change in pull request #4562: [WIP][AIRFLOW-3735] Separated parsing from checking

[GitBox]

ffinfo commented on a change in pull request #4562: [WIP][AIRFLOW-3735] 
Separated parsing from checking
URL: https://github.com/apache/airflow/pull/4562#discussion_r249584006
 
 

 ##
 File path: airflow/utils/dag_processing.py
 ##
 @@ -43,16 +46,18 @@
 
 # To avoid circular imports
 import airflow.models
-from airflow import configuration as conf
+from airflow import configuration as conf, configuration
 from airflow.dag.base_dag import BaseDag, BaseDagBag
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, AirflowDagCycleException
 from airflow.models import errors
 from airflow.settings import logging_class_path
 from airflow.utils import timezone
 from airflow.utils.db import provide_session
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow.utils.state import State
+from airflow.utils.timeout import timeout
 
+_log = logging.root.getChild(__package__)
 
 Review comment:
   Can I also use this when it's not a class? Or should I make a class with 
just a static method then?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ning-github opened a new pull request #4570: [AIRFLOW-XXX] Fix typos

[GitBox]

ning-github opened a new pull request #4570: [AIRFLOW-XXX] Fix typos
URL: https://github.com/apache/airflow/pull/4570
 
 
   Correct some spelling, grammar, and punctuation
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes: 
   
   Correct typos by:
   - removing periods from section headings for consistency and correctness
   - fixing some spelling
   - adding/editing for missing words
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non 
string types in Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#discussion_r249582548
 
 

 ##
 File path: tests/models.py
 ##
 @@ -408,6 +412,108 @@ def test_render_template_field(self):
 result = task.render_template('', '{{ foo }}', dict(foo='bar'))
 self.assertEqual(result, 'bar')
 
+def test_render_template_list_field(self):
+"""Tests if render_template from a list field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', ['{{ foo }}_1', '{{ foo }}_2'], 
dict(foo='bar'))
+assert_that(result, is_(instance_of(list)))
+assert_that(result, contains('bar_1', 'bar_2'))
+
+def test_render_template_tuple_field(self):
+"""Tests if render_template from a tuple field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', ('{{ foo }}_1', '{{ foo }}_2'), 
dict(foo='bar'))
+# tuple is replaced by a list
+assert_that(result, is_(instance_of(list)))
+assert_that(result, contains('bar_1', 'bar_2'))
+
+def test_render_template_dict_field(self):
+"""Tests if render_template from a dict field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', {'key1': '{{ foo }}_1', 'key2': '{{ 
foo }}_2'}, dict(foo='bar'))
+assert_that(result, is_({'key1': 'bar_1', 'key2': 'bar_2'}))
+
+def test_render_template_dict_field_with_templated_keys(self):
+"""Tests if render_template from a dict field works as expected:
+dictionary keys are not templated"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', {'key_{{ foo }}_1': 1, 'key_2': '{{ 
foo }}_2'}, dict(foo='bar'))
+assert_that(result, is_({'key_{{ foo }}_1': 1, 'key_2': 'bar_2'}))
+
+def test_render_template_date_field(self):
+"""Tests if render_template from a date field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+assert_that(
+task.render_template('', datetime.date(2018, 12, 6), 
dict(foo='bar')),
+is_(datetime.date(2018, 12, 6)))
+
+def test_render_template_datetime_field(self):
+"""Tests if render_template from a datetime field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+assert_that(
+task.render_template('', datetime.datetime(2018, 12, 6, 10, 55), 
dict(foo='bar')),
+is_(datetime.datetime(2018, 12, 6, 10, 55)))
+
+def test_render_template_UUID_field(self):
+"""Tests if render_template from a UUID field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+random_uuid = uuid.uuid4()
+assert_that(task.render_template('', random_uuid, dict(foo='bar')), 
is_(random_uuid))
 
 Review comment:
   ```suggestion
   assert_that(task.render_template('', random_uuid, {'foo':'bar'}), 
is_(random_uuid))
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non 
string types in Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#discussion_r249582636
 
 

 ##
 File path: tests/models.py
 ##
 @@ -408,6 +412,108 @@ def test_render_template_field(self):
 result = task.render_template('', '{{ foo }}', dict(foo='bar'))
 self.assertEqual(result, 'bar')
 
+def test_render_template_list_field(self):
+"""Tests if render_template from a list field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', ['{{ foo }}_1', '{{ foo }}_2'], 
dict(foo='bar'))
+assert_that(result, is_(instance_of(list)))
+assert_that(result, contains('bar_1', 'bar_2'))
+
+def test_render_template_tuple_field(self):
+"""Tests if render_template from a tuple field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', ('{{ foo }}_1', '{{ foo }}_2'), 
dict(foo='bar'))
+# tuple is replaced by a list
+assert_that(result, is_(instance_of(list)))
+assert_that(result, contains('bar_1', 'bar_2'))
+
+def test_render_template_dict_field(self):
+"""Tests if render_template from a dict field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', {'key1': '{{ foo }}_1', 'key2': '{{ 
foo }}_2'}, dict(foo='bar'))
+assert_that(result, is_({'key1': 'bar_1', 'key2': 'bar_2'}))
+
+def test_render_template_dict_field_with_templated_keys(self):
+"""Tests if render_template from a dict field works as expected:
+dictionary keys are not templated"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+result = task.render_template('', {'key_{{ foo }}_1': 1, 'key_2': '{{ 
foo }}_2'}, dict(foo='bar'))
+assert_that(result, is_({'key_{{ foo }}_1': 1, 'key_2': 'bar_2'}))
+
+def test_render_template_date_field(self):
+"""Tests if render_template from a date field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+assert_that(
+task.render_template('', datetime.date(2018, 12, 6), 
dict(foo='bar')),
+is_(datetime.date(2018, 12, 6)))
+
+def test_render_template_datetime_field(self):
+"""Tests if render_template from a datetime field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+assert_that(
+task.render_template('', datetime.datetime(2018, 12, 6, 10, 55), 
dict(foo='bar')),
+is_(datetime.datetime(2018, 12, 6, 10, 55)))
+
+def test_render_template_UUID_field(self):
+"""Tests if render_template from a UUID field works"""
+
+dag = DAG('test-dag',
+  start_date=DEFAULT_DATE)
+
+with dag:
+task = DummyOperator(task_id='op1')
+
+random_uuid = uuid.uuid4()
+assert_that(task.render_template('', random_uuid, dict(foo='bar')), 
is_(random_uuid))
 
 Review comment:
   Elsewhere too - please us `{}` rather than `dict()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

ashb commented on a change in pull request #4292: [AIRFLOW-2508] Handle non 
string types in Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#discussion_r249582407
 
 

 ##
 File path: tests/models.py
 ##
 @@ -31,10 +31,14 @@
 import time
 import unittest
 import urllib
+import uuid
 from tempfile import NamedTemporaryFile, mkdtemp
 
 import pendulum
 import six
+from hamcrest import contains, instance_of
+from hamcrest.core import assert_that
+from hamcrest.core.core import is_
 
 Review comment:
   Please just use the built in assert methods `assertIsIn`, 
`assertIsInstanceOf` etc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

ashb commented on issue #4292: [AIRFLOW-2508] Handle non string types in 
Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#issuecomment-456202545
 
 
   Could you give an example where this is used in a DAG so we can see it in 
context?
   
   I also wonder if it is worth noting something about this in the docs?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2508) Handle non string types in render_template_from_field

[Galak (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748225#comment-16748225
 ] 

Galak commented on AIRFLOW-2508:


[Pull Request #4292|https://github.com/apache/incubator-airflow/pull/4292] is 
waiting for a code review.

> Handle non string types in render_template_from_field
> -
>
> Key: AIRFLOW-2508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2508
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: models
>Affects Versions: 2.0.0
>Reporter: Eugene Brown
>Assignee: Galak
>Priority: Minor
>  Labels: easyfix, newbie
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> The render_template_from_field method of the BaseOperator class raises an 
> exception when it encounters content that is not a string_type, list, tuple 
> or dict.
> Example exception:
> {noformat}
> airflow.exceptions.AirflowException: Type '' used for parameter 
> 'job_flow_overrides[Instances][InstanceGroups][InstanceCount]' is not 
> supported for templating{noformat}
> I propose instead that when it encounters content of other types it returns 
> the content unchanged, rather than raising an exception.
> Consider this case: I extended the EmrCreateJobFlowOperator to make the 
> job_flow_overrides argument a templatable field. job_flow_overrides is a 
> dictionary with a mix of strings, integers and booleans for values.
> When I extended the class as such:
> {code:java}
> class EmrCreateJobFlowOperatorTemplateOverrides(EmrCreateJobFlowOperator):
> template_fields = ['job_flow_overrides']{code}
> And added a task to my dag with this format:
> {code:java}
> step_create_cluster = EmrCreateJobFlowOperatorTemplateOverrides(
> task_id="create_cluster",
> job_flow_overrides={
> "Name": "my-cluster {{ dag_run.conf['run_date'] }}",
> "Instances": {
> "InstanceGroups": [
> {
> "Name": "Master nodes",
> "InstanceType": "c3.4xlarge",
> "InstanceCount": 1
> },
> {
> "Name": "Slave nodes",
> "InstanceType": "c3.4xlarge",
> "InstanceCount": 4
> },
> "TerminationProtected": False
> ]
> },
> "BootstrapActions": [{
>  "Name": "Custom action",
>  "ScriptBootstrapAction": {
>  "Path": "s3://repo/{{ dag_run.conf['branch'] 
> }}/requirements.txt"
>  }
> }],
>},
>aws_conn_id='aws_default',
>emr_conn_id='aws_default',
>dag=dag
> )
> {code}
> The exception I gave above was raised and the step failed. I think it would 
> be preferable for the method to instead pass over numeric and boolean values 
> as users may want to use template_fields in the way I have to template string 
> values in dictionaries or lists of mixed types.
> Here is the render_template_from_field method from the BaseOperator:
> {code:java}
> def render_template_from_field(self, attr, content, context, jinja_env):
> """
> Renders a template from a field. If the field is a string, it will
> simply render the string and return the result. If it is a collection or
> nested set of collections, it will traverse the structure and render
> all strings in it.
> """
> rt = self.render_template
> if isinstance(content, six.string_types):
> result = jinja_env.from_string(content).render(**context)
> elif isinstance(content, (list, tuple)):
> result = [rt(attr, e, context) for e in content]
> elif isinstance(content, dict):
> result = {
> k: rt("{}[{}]".format(attr, k), v, context)
> for k, v in list(content.items())}
> else:
> param_type = type(content)
> msg = (
> "Type '{param_type}' used for parameter '{attr}' is "
> "not supported for templating").format(**locals())
> raise AirflowException(msg)
> return result{code}
>  I propose that the method returns content unchanged if the content is of one 
> of (int, float, complex, bool) types. So my solution would include an extra 
> elif in the form:
> {code}
> elif isinstance(content, (int, float, complex, bool)):
> result = content
> {code}
>  Are there any reasons this would be a bad idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] galak75 commented on issue #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

galak75 commented on issue #4292: [AIRFLOW-2508] Handle non string types in 
Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#issuecomment-456200860
 
 
   I rebased my PR onto master. Is there something more I should do so that 
this PR can be reviewed ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

codecov-io edited a comment on issue #4292: [AIRFLOW-2508] Handle non string 
types in Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#issuecomment-455914575
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=h1) 
Report
   > Merging 
[#4292](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d7ece3da6edd71007f70a0ccf5dc1a9f9d28f86?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4292/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4292  +/-   ##
   ==
   + Coverage74.1%   74.13%   +0.02% 
   ==
 Files 421  421  
 Lines   2768127695  +14 
   ==
   + Hits2051420531  +17 
   + Misses   7167 7164   -3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.5% <100%> (+0.16%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5)
 | `75.75% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `81.39% <0%> (+3.78%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=footer). 
Last update 
[0d7ece3...77e30bc](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2009) DataFlowHook does not use correct service account

[Kaxil Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-2009.
-
   Resolution: Fixed
Fix Version/s: 1.10.3

> DataFlowHook does not use correct service account
> -
>
> Key: AIRFLOW-2009
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2009
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: Dataflow, hooks
>Affects Versions: 2.0.0
>Reporter: Jessica Laughlin
>Assignee: Feng Lu
>Priority: Major
> Fix For: 1.10.3
>
>
> We have been using the DataFlowOperator to schedule DataFlow jobs.
> We found that the DataFlowHook used by the DataFlowOperator doesn't actually 
> use the passed `gcp_conn_id` to schedule the DataFlow job, but only to read 
> the results after. 
> code 
> (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/gcp_dataflow_hook.py#L158):
> _Dataflow(cmd).wait_for_done()
> _DataflowJob(self.get_conn(), variables['project'],
>  name, self.poll_sleep).wait_for_done()
> The first line here should also be using self.get_conn(). 
> For this reason, our tasks using the DataFlowOperator have actually been 
> using the default Google Compute Engine service account (which has DataFlow 
> permissions) to schedule DataFlow jobs. It is only when our provided service 
> account (which does not have DataFlow permissions) is used in the second line 
> that we are seeing a permissions error. 
> I would like to fix this bug, but have to work around it at the moment due to 
> time constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2009) DataFlowHook does not use correct service account

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748202#comment-16748202
 ] 

ASF subversion and git services commented on AIRFLOW-2009:
--

Commit 5084305e31072fc06ae3415e7e5f4be755e1d5a7 in airflow's branch 
refs/heads/master from fenglu-g
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5084305 ]

[AIRFLOW-2009] Fix dataflow hook connection-id (#4563)



> DataFlowHook does not use correct service account
> -
>
> Key: AIRFLOW-2009
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2009
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: Dataflow, hooks
>Affects Versions: 2.0.0
>Reporter: Jessica Laughlin
>Assignee: Feng Lu
>Priority: Major
>
> We have been using the DataFlowOperator to schedule DataFlow jobs.
> We found that the DataFlowHook used by the DataFlowOperator doesn't actually 
> use the passed `gcp_conn_id` to schedule the DataFlow job, but only to read 
> the results after. 
> code 
> (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/gcp_dataflow_hook.py#L158):
> _Dataflow(cmd).wait_for_done()
> _DataflowJob(self.get_conn(), variables['project'],
>  name, self.poll_sleep).wait_for_done()
> The first line here should also be using self.get_conn(). 
> For this reason, our tasks using the DataFlowOperator have actually been 
> using the default Google Compute Engine service account (which has DataFlow 
> permissions) to schedule DataFlow jobs. It is only when our provided service 
> account (which does not have DataFlow permissions) is used in the second line 
> that we are seeing a permissions error. 
> I would like to fix this bug, but have to work around it at the moment due to 
> time constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2009) DataFlowHook does not use correct service account

[Kaxil Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-2009:

Fix Version/s: 2.0.0

> DataFlowHook does not use correct service account
> -
>
> Key: AIRFLOW-2009
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2009
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: Dataflow, hooks
>Affects Versions: 2.0.0
>Reporter: Jessica Laughlin
>Assignee: Feng Lu
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
>
> We have been using the DataFlowOperator to schedule DataFlow jobs.
> We found that the DataFlowHook used by the DataFlowOperator doesn't actually 
> use the passed `gcp_conn_id` to schedule the DataFlow job, but only to read 
> the results after. 
> code 
> (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/gcp_dataflow_hook.py#L158):
> _Dataflow(cmd).wait_for_done()
> _DataflowJob(self.get_conn(), variables['project'],
>  name, self.poll_sleep).wait_for_done()
> The first line here should also be using self.get_conn(). 
> For this reason, our tasks using the DataFlowOperator have actually been 
> using the default Google Compute Engine service account (which has DataFlow 
> permissions) to schedule DataFlow jobs. It is only when our provided service 
> account (which does not have DataFlow permissions) is used in the second line 
> that we are seeing a permissions error. 
> I would like to fix this bug, but have to work around it at the moment due to 
> time constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil commented on issue #4563: [AIRFLOW-2009] Fix dataflow hook connection-id

[GitBox]

kaxil commented on issue #4563: [AIRFLOW-2009] Fix dataflow hook connection-id
URL: https://github.com/apache/airflow/pull/4563#issuecomment-456188818
 
 
   Thanks @fenglu-g 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2009) DataFlowHook does not use correct service account

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748201#comment-16748201
 ] 

ASF GitHub Bot commented on AIRFLOW-2009:
-

kaxil commented on pull request #4563: [AIRFLOW-2009] Fix dataflow hook 
connection-id
URL: https://github.com/apache/airflow/pull/4563
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DataFlowHook does not use correct service account
> -
>
> Key: AIRFLOW-2009
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2009
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: Dataflow, hooks
>Affects Versions: 2.0.0
>Reporter: Jessica Laughlin
>Assignee: Feng Lu
>Priority: Major
>
> We have been using the DataFlowOperator to schedule DataFlow jobs.
> We found that the DataFlowHook used by the DataFlowOperator doesn't actually 
> use the passed `gcp_conn_id` to schedule the DataFlow job, but only to read 
> the results after. 
> code 
> (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/gcp_dataflow_hook.py#L158):
> _Dataflow(cmd).wait_for_done()
> _DataflowJob(self.get_conn(), variables['project'],
>  name, self.poll_sleep).wait_for_done()
> The first line here should also be using self.get_conn(). 
> For this reason, our tasks using the DataFlowOperator have actually been 
> using the default Google Compute Engine service account (which has DataFlow 
> permissions) to schedule DataFlow jobs. It is only when our provided service 
> account (which does not have DataFlow permissions) is used in the second line 
> that we are seeing a permissions error. 
> I would like to fix this bug, but have to work around it at the moment due to 
> time constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil merged pull request #4563: [AIRFLOW-2009] Fix dataflow hook connection-id

[GitBox]

kaxil merged pull request #4563: [AIRFLOW-2009] Fix dataflow hook connection-id
URL: https://github.com/apache/airflow/pull/4563
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4563: [AIRFLOW-2009] Fix dataflow hook connection-id

[GitBox]

codecov-io edited a comment on issue #4563: [AIRFLOW-2009] Fix dataflow hook 
connection-id
URL: https://github.com/apache/airflow/pull/4563#issuecomment-456181017
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=h1) 
Report
   > Merging 
[#4563](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/39dab6f4019f8107814104257274f3c7e78efeaa?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `95%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4563/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4563  +/-   ##
   ==
   + Coverage   74.11%   74.12%   +0.01% 
   ==
 Files 421  421  
 Lines   2768127700  +19 
   ==
   + Hits2051520532  +17 
   - Misses   7166 7168   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4563/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5)
 | `75.75% <100%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/4563/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `81.39% <94.73%> (+3.78%)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4563/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.33% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=footer). 
Last update 
[39dab6f...de08a2f](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-1262) Allow configuration of email alert subject and body

[Kaxil Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-1262.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Allow configuration of email alert subject and body
> ---
>
> Key: AIRFLOW-1262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1262
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: configuration
>Reporter: Alek Storm
>Assignee: Alek Storm
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1262) Allow configuration of email alert subject and body

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748185#comment-16748185
 ] 

ASF GitHub Bot commented on AIRFLOW-1262:
-

kaxil commented on pull request #4557: [AIRFLOW-1262] Add missing doc for email 
configuration
URL: https://github.com/apache/airflow/pull/4557
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow configuration of email alert subject and body
> ---
>
> Key: AIRFLOW-1262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1262
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: configuration
>Reporter: Alek Storm
>Assignee: Alek Storm
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil commented on issue #4557: [AIRFLOW-1262] Add missing doc for email configuration

[GitBox]

kaxil commented on issue #4557: [AIRFLOW-1262] Add missing doc for email 
configuration
URL: https://github.com/apache/airflow/pull/4557#issuecomment-456182114
 
 
   Thanks @feluelle 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1262) Allow configuration of email alert subject and body

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748186#comment-16748186
 ] 

ASF subversion and git services commented on AIRFLOW-1262:
--

Commit 0d7ece3da6edd71007f70a0ccf5dc1a9f9d28f86 in airflow's branch 
refs/heads/master from Felix
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0d7ece3 ]

[AIRFLOW-1262] Adds missing docs for email configuration (#4557)



> Allow configuration of email alert subject and body
> ---
>
> Key: AIRFLOW-1262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1262
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: configuration
>Reporter: Alek Storm
>Assignee: Alek Storm
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil merged pull request #4557: [AIRFLOW-1262] Add missing doc for email configuration

[GitBox]

kaxil merged pull request #4557: [AIRFLOW-1262] Add missing doc for email 
configuration
URL: https://github.com/apache/airflow/pull/4557
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3732) Fix issue when trying to edit connection in RBAC UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748182#comment-16748182
 ] 

ASF GitHub Bot commented on AIRFLOW-3732:
-

kaxil commented on pull request #4559: [AIRFLOW-3732] Fix issue when editing 
connection in RBAC UI
URL: https://github.com/apache/airflow/pull/4559
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix issue when trying to edit connection in RBAC UI
> ---
>
> Key: AIRFLOW-3732
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3732
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.2
>
>
> Error without fix:
> {code}
> Traceback (most recent call last):
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1982, in wsgi_app
> response = self.full_dispatch_request()
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1612, in full_dispatch_request
> rv = self.dispatch_request()
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py",
>  line 26, in wraps
> return f(self, *args, **kwargs)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/views.py",
>  line 531, in edit
> related_views=self._related_views)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/baseviews.py",
>  line 160, in render_template
> return render_template(template, **dict(list(kwargs.items()) + 
> list(self.extra_args.items(
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/templating.py",
>  line 134, in render_template
> context, ctx.app)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/templating.py",
>  line 116, in _render
> rv = template.render(context)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/jinja2/environment.py",
>  line 1008, in render
> return self.environment.handle_exception(exc_info, True)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/jinja2/environment.py",
>  line 780, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/airflow/conn_edit.html",
>  line 18, in top-level template code
> {% extends 'appbuilder/general/model/edit.html' %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html",
>  line 2, in top-level template code
> {% import 'appbuilder/general/lib.html' as lib %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/base.html",
>  line 1, in top-level template code
> {% extends base_template %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/appbuilder/baselayout.html",
>  line 19, in top-level template code
> {% import 'appbuilder/baselib.html' as baselib %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/init.html",
>  line 60, in top-level template code
> {% block tail %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/airflow/conn_edit.html",
>  line 22, in block "tail"
> 
> TypeError: get_asset_url() got multiple values for keyword argument 'filename'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3732) Fix issue when trying to edit connection in RBAC UI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748183#comment-16748183
 ] 

ASF subversion and git services commented on AIRFLOW-3732:
--

Commit fea29afb65f6685ad4fe38935d970c8cdd79 in airflow's branch 
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fea29af ]

[AIRFLOW-3732] Fix issue when trying to edit connection in RBAC UI (#4559)



> Fix issue when trying to edit connection in RBAC UI
> ---
>
> Key: AIRFLOW-3732
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3732
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.2
>
>
> Error without fix:
> {code}
> Traceback (most recent call last):
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1982, in wsgi_app
> response = self.full_dispatch_request()
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1612, in full_dispatch_request
> rv = self.dispatch_request()
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/app.py",
>  line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py",
>  line 26, in wraps
> return f(self, *args, **kwargs)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/views.py",
>  line 531, in edit
> related_views=self._related_views)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/baseviews.py",
>  line 160, in render_template
> return render_template(template, **dict(list(kwargs.items()) + 
> list(self.extra_args.items(
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/templating.py",
>  line 134, in render_template
> context, ctx.app)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask/templating.py",
>  line 116, in _render
> rv = template.render(context)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/jinja2/environment.py",
>  line 1008, in render
> return self.environment.handle_exception(exc_info, True)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/jinja2/environment.py",
>  line 780, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/airflow/conn_edit.html",
>  line 18, in top-level template code
> {% extends 'appbuilder/general/model/edit.html' %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html",
>  line 2, in top-level template code
> {% import 'appbuilder/general/lib.html' as lib %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/base.html",
>  line 1, in top-level template code
> {% extends base_template %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/appbuilder/baselayout.html",
>  line 19, in top-level template code
> {% import 'appbuilder/baselib.html' as baselib %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/flask_appbuilder/templates/appbuilder/init.html",
>  line 60, in top-level template code
> {% block tail %}
>   File 
> "/Users/kaxil/.virtualenvs/airflow_1.10.2rc2/lib/python2.7/site-packages/airflow/www_rbac/templates/airflow/conn_edit.html",
>  line 22, in block "tail"
> 
> TypeError: get_asset_url() got multiple values for keyword argument 'filename'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil merged pull request #4565: [AIRFLOW-XXX] Fix doc string DataprocClusterDeleteOperator

[GitBox]

kaxil merged pull request #4565: [AIRFLOW-XXX] Fix doc string 
DataprocClusterDeleteOperator
URL: https://github.com/apache/airflow/pull/4565
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil merged pull request #4559: [AIRFLOW-3732] Fix issue when editing connection in RBAC UI

[GitBox]

kaxil merged pull request #4559: [AIRFLOW-3732] Fix issue when editing 
connection in RBAC UI
URL: https://github.com/apache/airflow/pull/4559
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4563: [AIRFLOW-2009] Fix dataflow hook connection-id

[GitBox]

codecov-io commented on issue #4563: [AIRFLOW-2009] Fix dataflow hook 
connection-id
URL: https://github.com/apache/airflow/pull/4563#issuecomment-456181017
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=h1) 
Report
   > Merging 
[#4563](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/39dab6f4019f8107814104257274f3c7e78efeaa?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `95%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4563/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4563  +/-   ##
   ==
   + Coverage   74.11%   74.12%   +0.01% 
   ==
 Files 421  421  
 Lines   2768127700  +19 
   ==
   + Hits2051520533  +18 
   - Misses   7166 7167   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4563/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5)
 | `75.75% <100%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/4563/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `81.39% <94.73%> (+3.78%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=footer). 
Last update 
[39dab6f...de08a2f](https://codecov.io/gh/apache/airflow/pull/4563?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4567: [AIRFLOW-3742] Respect the `fallback` arg in airflow.configuration.get

[GitBox]

codecov-io commented on issue #4567: [AIRFLOW-3742] Respect the `fallback` arg 
in airflow.configuration.get
URL: https://github.com/apache/airflow/pull/4567#issuecomment-456178255
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=h1) 
Report
   > Merging 
[#4567](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/39dab6f4019f8107814104257274f3c7e78efeaa?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4567/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4567  +/-   ##
   ==
   + Coverage   74.11%   74.12%   +0.01% 
   ==
 Files 421  421  
 Lines   2768127675   -6 
   ==
   - Hits2051520514   -1 
   + Misses   7166 7161   -5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/4567/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `80.88% <100%> (+0.6%)` | :arrow_up: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4567/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `92.08% <100%> (ø)` | :arrow_up: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/4567/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `77.65% <0%> (+0.27%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=footer). 
Last update 
[39dab6f...983dca2](https://codecov.io/gh/apache/airflow/pull/4567?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3745) Fix viewer not able to view dag details

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748171#comment-16748171
 ] 

ASF GitHub Bot commented on AIRFLOW-3745:
-

feng-tao commented on pull request #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3745
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   To stick with existing RBAC which Admin, Op, User, Viewer view all dags.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix viewer not able to view dag details
> ---
>
> Key: AIRFLOW-3745
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3745
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao opened a new pull request #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

[GitBox]

feng-tao opened a new pull request #4569: [AIRFLOW-3745] Fix viewer not able to 
view dag details
URL: https://github.com/apache/airflow/pull/4569
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3745
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   To stick with existing RBAC which Admin, Op, User, Viewer view all dags.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] fenglu-g commented on a change in pull request #4563: [AIRFLOW-2009] Fix dataflow hook connection-id

[GitBox]

fenglu-g commented on a change in pull request #4563: [AIRFLOW-2009] Fix 
dataflow hook connection-id
URL: https://github.com/apache/airflow/pull/4563#discussion_r249553189
 
 

 ##
 File path: airflow/contrib/hooks/gcp_api_base_hook.py
 ##
 @@ -24,13 +24,16 @@
 import google.auth
 import google_auth_httplib2
 import google.oauth2.service_account
+import os
+import tempfile
 
 from airflow.exceptions import AirflowException
 from airflow.hooks.base_hook import BaseHook
 from airflow.utils.log.logging_mixin import LoggingMixin
 
 
 _DEFAULT_SCOPES = ('https://www.googleapis.com/auth/cloud-platform',)
+G_APP_CRED = "GOOGLE_APPLICATION_CREDENTIALS"
 
 Review comment:
   Done. On a second thought, I am going to make this private too as it's only 
used in gcp_api_base_hook. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3745) Fix viewer not able to view dag details

[Tao Feng (JIRA)]

Tao Feng created AIRFLOW-3745:
-

 Summary: Fix viewer not able to view dag details
 Key: AIRFLOW-3745
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3745
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Tao Feng
Assignee: Tao Feng






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] mik-laj commented on a change in pull request #4562: [WIP][AIRFLOW-3735] Separated parsing from checking

[GitBox]

mik-laj commented on a change in pull request #4562: [WIP][AIRFLOW-3735] 
Separated parsing from checking
URL: https://github.com/apache/airflow/pull/4562#discussion_r249540287
 
 

 ##
 File path: airflow/utils/dag_processing.py
 ##
 @@ -43,16 +46,18 @@
 
 # To avoid circular imports
 import airflow.models
-from airflow import configuration as conf
+from airflow import configuration as conf, configuration
 from airflow.dag.base_dag import BaseDag, BaseDagBag
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, AirflowDagCycleException
 from airflow.models import errors
 from airflow.settings import logging_class_path
 from airflow.utils import timezone
 from airflow.utils.db import provide_session
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow.utils.state import State
+from airflow.utils.timeout import timeout
 
+_log = logging.root.getChild(__package__)
 
 Review comment:
   I think, you should use `LoggingMixin'.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3744) Usage of deprecated method in Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748102#comment-16748102
 ] 

ASF GitHub Bot commented on AIRFLOW-3744:
-

mik-laj commented on pull request #4568: [AIRFLOW-3744] Abandon the use of 
obsolete aliases of methods
URL: https://github.com/apache/airflow/pull/4568
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-3744
   
   ### Description
   
   New versions of Python have marked some methods as obsolete. This change 
replaces valid calls.
   https://docs.python.org/3/library/unittest.html#deprecated-aliases
   
   ### Tests
   
   Not applicable.
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   Not applicable.
   
   ### Code Quality
   
   - [X] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Usage of deprecated method in Unit Tests
> 
>
> Key: AIRFLOW-3744
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3744
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> Some tests use a deprecated alias: 
> [https://docs.python.org/3/library/unittest.html#deprecated-aliases]
> These tests should stop doing this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] mik-laj opened a new pull request #4568: [AIRFLOW-3744] Abandon the use of obsolete aliases of methods

[GitBox]

mik-laj opened a new pull request #4568: [AIRFLOW-3744] Abandon the use of 
obsolete aliases of methods
URL: https://github.com/apache/airflow/pull/4568
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-3744
   
   ### Description
   
   New versions of Python have marked some methods as obsolete. This change 
replaces valid calls.
   https://docs.python.org/3/library/unittest.html#deprecated-aliases
   
   ### Tests
   
   Not applicable.
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   Not applicable.
   
   ### Code Quality
   
   - [X] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3744) Usage of deprecated method in Unit Tests

[Kamil Bregula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula reassigned AIRFLOW-3744:
--

Assignee: Kamil Bregula

> Usage of deprecated method in Unit Tests
> 
>
> Key: AIRFLOW-3744
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3744
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> Some tests use a deprecated alias: 
> [https://docs.python.org/3/library/unittest.html#deprecated-aliases]
> These tests should stop doing this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3744) Usage of deprecated method in Unit Tests

[Kamil Bregula (JIRA)]

Kamil Bregula created AIRFLOW-3744:
--

 Summary: Usage of deprecated method in Unit Tests
 Key: AIRFLOW-3744
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3744
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Kamil Bregula


Some tests use a deprecated alias: 
[https://docs.python.org/3/library/unittest.html#deprecated-aliases]

These tests should stop doing this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4516: [AIRFLOW-3719] Handle StopIteration in CloudWatch logs retrieval

[GitBox]

codecov-io edited a comment on issue #4516: [AIRFLOW-3719] Handle StopIteration 
in CloudWatch logs retrieval
URL: https://github.com/apache/airflow/pull/4516#issuecomment-453944867
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=h1) 
Report
   > Merging 
[#4516](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0949cbd4d9e57abf9d4c2f439d34b9703cb9a05?src=pr=desc)
 will **increase** coverage by `1.36%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4516/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4516  +/-   ##
   ==
   + Coverage   73.81%   75.18%   +1.36% 
   ==
 Files 421  421  
 Lines   2766229624+1962 
   ==
   + Hits2041822272+1854 
   - Misses   7244 7352 +108
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/sagemaker\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NhZ2VtYWtlcl9ob29rLnB5)
 | `88.21% <100%> (+0.36%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/datadog\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2RhdGFkb2dfaG9vay5weQ==)
 | `77.55% <0%> (-22.45%)` | :arrow_down: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `87.75% <0%> (-4.36%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5)
 | `69.81% <0%> (-3.88%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `77.87% <0%> (-1.87%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/gcp\_function\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9mdW5jdGlvbl9ob29rLnB5)
 | `71.42% <0%> (-1.65%)` | :arrow_down: |
   | 
[airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5)
 | `96.1% <0%> (-0.2%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.41% <0%> (+0.67%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `79.03% <0%> (+0.8%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_sql\_hook.py](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9zcWxfaG9vay5weQ==)
 | `70.69% <0%> (+1.48%)` | :arrow_up: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/4516/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=footer). 
Last update 
[c0949cb...c90b64d](https://codecov.io/gh/apache/airflow/pull/4516?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3742) airflow.configuration doesn't respect fallback kwarg

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748092#comment-16748092
 ] 

ASF subversion and git services commented on AIRFLOW-3742:
--

Commit 983dca2e326b7246f8b063ff3de8f559185c18d9 in airflow's branch 
refs/heads/config-parser-fallback-get from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=983dca2 ]

[AIRFLOW-3742] Respect the `fallback` arg in airflow.configuration.get

This argument is part of the API from our parent class, but we didn't
support it because of the various steps we perform in `get()` - this
makes it behave more like the parent class, and can simplify a few
instances in our code (I've only included one that I found here)


> airflow.configuration doesn't respect fallback kwarg
> 
>
> Key: AIRFLOW-3742
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3742
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.3
>
>
> The ConfigParser class we base our config parser on can optionally accept a 
> fallback argument:
> {code:python}
> conf.get('sect', 'key', fallback='default')
> {code}
> but our sub-class doesn't support this. It would be nice if we did. Then we 
> can turn
> {code:python}
> plugins_folder = configuration.conf.get('core', 'plugins_folder')
> if not plugins_folder:
> plugins_folder = settings.AIRFLOW_HOME + '/plugins'
> {code}
> in to this:
> {code:python}
> plugins_folder = configuration.conf.get('core', 'plugins_folder', 
> fallback=settings.AIRFLOW_HOME + '/plugins')
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3743) Unify multiple ways of specifying and accessing AIRFLOW_HOME

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748083#comment-16748083
 ] 

ASF subversion and git services commented on AIRFLOW-3743:
--

Commit 66d016c84b06167b72600a1e60270da684852c60 in airflow's branch 
refs/heads/unify_airflow_home from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=66d016c ]

[AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME

There were a few ways of getting the AIRFLOW_HOME directory used
throughout the code base, giving possibly conflicting answer if they
weren't kept in sync:

- the AIRFLOW_HOME environment variable
- core/airflow_home from the config
- settings.AIRFLOW_HOME
- configuration.AIRFLOW_HOME

Since the home directory is used to compute the default path of the
config file to load, specifying the home directory Again in the config
file didn't make any sense to me, and I have deprecated that.

This commit makes everything in the code base use
`settings.AIRFLOW_HOME` as the source of truth, and deprecates the
core/airflow_home config option.

(This issue caused me a problem where the RBAC UI wouldn't work as it
didn't find the right webserver_config.py)


> Unify multiple ways of specifying and accessing AIRFLOW_HOME
> 
>
> Key: AIRFLOW-3743
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3743
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.3
>
>
> We currently have two (or three, depending on how you look at it) ways of 
> getting AIRFLOW_HOME
> {{airflow.configuration.AIRFLOW_HOME}}
> {{airflow.settings.AIRFLOW_HOME}}
> {{airflow.configuration.get('core', 'airflow_home')}}
> used throughout the codebase. This has led to a few odd bugs 
> (https://github.com/puckel/docker-airflow/issues/225 for example) if the 
> config file and environment are out-of-sync.
> Since the default path of the config file is dependent on the AIRFLOW_HOME 
> environment variable I propose we remove the airflow_home from the config 
> file, and unify all access in the code base to go via 
> {{settings.AIRFLOW_HOME}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io commented on issue #4566: [AIRFLOW-3566] - Adding timeout and login_timeout to MsSqlHook

[GitBox]

codecov-io commented on issue #4566: [AIRFLOW-3566] - Adding timeout and 
login_timeout to MsSqlHook
URL: https://github.com/apache/airflow/pull/4566#issuecomment-456133118
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=h1) 
Report
   > Merging 
[#4566](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/39dab6f4019f8107814104257274f3c7e78efeaa?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4566/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4566  +/-   ##
   =
   - Coverage   74.11%   74.1%   -0.01% 
   =
 Files 421 421  
 Lines   27681   27683   +2 
   =
 Hits20515   20515  
   - Misses   71667168   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/4566/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5)
 | `0% <0%> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=footer). 
Last update 
[39dab6f...ecf87d6](https://codecov.io/gh/apache/airflow/pull/4566?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

XD-DENG commented on a change in pull request #4514: [AIRFLOW-3698] Add 
documentation for AWS Connection
URL: https://github.com/apache/airflow/pull/4514#discussion_r249514435
 
 

 ##
 File path: docs/howto/manage-connections.rst
 ##
 @@ -151,6 +151,52 @@ Scopes (comma separated)
 issue `AIRFLOW-2522
 `_.
 
+Amazon Web Services
+~~~
+
+The Amazon Web Services connection type enables the :ref:`AWS Integrations
+`.
+
+Authenticating to AWS
+'
+
+Authentication may be performed using any of the `boto3 options 
`_.
 Alternatively, one can pass credentials in as a Connection initialisation 
parameter.
+
+Default Connection IDs
+''
+
+The default connection ID is ``aws_default``.
+
+Configuring the Connection
+''
+
+Login (optional)
+Specify the AWS access key ID.
+
+Password (optional)
+Specify the AWS secret access key.
+
+Extra (optional)
 
 Review comment:
   it may be good to mention that users can specify endpoint in `Extra` section 
by giving "host". This will be useful in cases like they're using S3-compatible 
services (but not AWS service.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

ashb commented on a change in pull request #4514: [AIRFLOW-3698] Add 
documentation for AWS Connection
URL: https://github.com/apache/airflow/pull/4514#discussion_r24952
 
 

 ##
 File path: docs/howto/manage-connections.rst
 ##
 @@ -151,6 +151,52 @@ Scopes (comma separated)
 issue `AIRFLOW-2522
 `_.
 
+Amazon Web Services
+~~~
+
+The Amazon Web Services connection type enables the :ref:`AWS Integrations
+`.
+
+Authenticating to AWS
+'
+
+Authentication may be performed using any of the `boto3 options 
`_.
 Alternatively, one can pass credentials in as a Connection initialisation 
parameter.
+
+Default Connection IDs
+''
+
+The default connection ID is ``aws_default``.
+
+Configuring the Connection
+''
+
+Login (optional)
+Specify the AWS access key ID.
+
+Password (optional)
+Specify the AWS secret access key.
+
+Extra (optional)
+Specify the extra parameters (as json dictionary) that can be used in ssh
+connection. The following parameters are supported:
+
+* **aws_access_key_id**: AWS access key ID of the connection. To use this, 
the `aws_secret_access_key` parameter must also be passed in.
+* **aws_secret_access_key**: AWS secret access key of the connection
 
 Review comment:
   I would rather we didn't document the key_id and access_key in extra, as 
those really belong in the login and password.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4514: [AIRFLOW-3698] Add documentation for AWS Connection

[GitBox]

ashb commented on a change in pull request #4514: [AIRFLOW-3698] Add 
documentation for AWS Connection
URL: https://github.com/apache/airflow/pull/4514#discussion_r249511900
 
 

 ##
 File path: docs/howto/manage-connections.rst
 ##
 @@ -151,6 +151,52 @@ Scopes (comma separated)
 issue `AIRFLOW-2522
 `_.
 
+Amazon Web Services
+~~~
+
+The Amazon Web Services connection type enables the :ref:`AWS Integrations
+`.
+
+Authenticating to AWS
+'
+
+Authentication may be performed using any of the `boto3 options 
`_.
 Alternatively, one can pass credentials in as a Connection initialisation 
parameter.
+
+Default Connection IDs
+''
+
+The default connection ID is ``aws_default``.
+
+Configuring the Connection
+''
+
+Login (optional)
+Specify the AWS access key ID.
+
+Password (optional)
+Specify the AWS secret access key.
+
+Extra (optional)
+Specify the extra parameters (as json dictionary) that can be used in ssh
 
 Review comment:
   `ssh` :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3742) airflow.configuration doesn't respect fallback kwarg

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748057#comment-16748057
 ] 

ASF GitHub Bot commented on AIRFLOW-3742:
-

ashb commented on pull request #4567: [AIRFLOW-3742] Respect the `fallback` arg 
in airflow.configuration.get
URL: https://github.com/apache/airflow/pull/4567
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] https://issues.apache.org/jira/browse/AIRFLOW-3742
   
   
   ### Description
   
   - [x] This `fallback` argument is part of the API from our parent class, but 
we didn't
   support it because of the various steps we perform in `get()` - this
   makes it behave more like the parent class, and can simplify a few
   instances in our code (I've only included one that I found here)
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: few small tests added
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> airflow.configuration doesn't respect fallback kwarg
> 
>
> Key: AIRFLOW-3742
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3742
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.3
>
>
> The ConfigParser class we base our config parser on can optionally accept a 
> fallback argument:
> {code:python}
> conf.get('sect', 'key', fallback='default')
> {code}
> but our sub-class doesn't support this. It would be nice if we did. Then we 
> can turn
> {code:python}
> plugins_folder = configuration.conf.get('core', 'plugins_folder')
> if not plugins_folder:
> plugins_folder = settings.AIRFLOW_HOME + '/plugins'
> {code}
> in to this:
> {code:python}
> plugins_folder = configuration.conf.get('core', 'plugins_folder', 
> fallback=settings.AIRFLOW_HOME + '/plugins')
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] mik-laj commented on a change in pull request #4524: [AIRFLOW-3707] Group subpackages by cloud providers

[GitBox]

mik-laj commented on a change in pull request #4524: [AIRFLOW-3707] Group 
subpackages by cloud providers
URL: https://github.com/apache/airflow/pull/4524#discussion_r249502406
 
 

 ##
 File path: setup.py
 ##
 @@ -211,27 +217,44 @@ def write_version(filename=os.path.join(*['airflow',
 qds = ['qds-sdk>=1.10.4']
 rabbitmq = ['librabbitmq>=1.6.1']
 redis = ['redis>=2.10.5,<3.0.0']
-s3 = ['boto3>=1.7.0, <1.8.0']
 salesforce = ['simple-salesforce>=0.72']
 samba = ['pysmbclient>=0.1.3']
 segment = ['analytics-python>=1.2.9']
 sendgrid = ['sendgrid>=5.2.0']
 slack = ['slackclient>=1.0.0']
 mongo = ['pymongo>=3.6.0']
-snowflake = ['snowflake-connector-python>=1.5.2',
- 'snowflake-sqlalchemy>=1.1.0']
-ssh = ['paramiko>=2.1.1', 'pysftp>=0.2.9', 'sshtunnel>=0.1.4,<0.2']
+snowflake = [
+'snowflake-connector-python>=1.5.2',
+'snowflake-sqlalchemy>=1.1.0',
+]
+ssh = [
+'paramiko>=2.1.1',
+'pysftp>=0.2.9',
+'sshtunnel>=0.1.4,<0.2',
+]
 statsd = ['statsd>=3.0.1, <4.0']
 vertica = ['vertica-python>=0.5.1']
 webhdfs = ['hdfs[dataframe,avro,kerberos]>=2.0.4']
 winrm = ['pywinrm==0.2.2']
 zendesk = ['zdesk']
 
-all_dbs = postgres + mysql + hive + mssql + hdfs + vertica + cloudant + druid 
+ pinot \
-+ cassandra + mongo
+all_dbs = (
+postgres +
+mysql +
+hive +
+mssql +
+hdfs +
+vertica +
+cloudant +
+druid +
+pinot +
+cassandra +
 
 Review comment:
   I'm done. I hope I did not break something.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3566) MsSqlHook timeout is not configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748040#comment-16748040
 ] 

ASF GitHub Bot commented on AIRFLOW-3566:
-

eladkal commented on pull request #4566: [AIRFLOW-3566] - Adding timeout and 
login_timeout to MsSqlHook
URL: https://github.com/apache/airflow/pull/4566
 
 
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3566) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3566
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   Adding timeout and login_timeout properties.
   timeout  - property of the query (query timeout in seconds).
   login_timeout -  property of connection (timeout for connection) allowing it 
to be set from Connection and overwrite in the hook.
   
   Default values (preserve backwards comparability):
   timeout default value is 0 (no timeout)
   login_timeout default value is 60
   
   
   ### Code Quality
   
   - [V] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> MsSqlHook timeout is not configurable
> -
>
> Key: AIRFLOW-3566
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3566
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Mark Zitnik
>Priority: Minor
> Fix For: 1.10.0
>
>
> I have looking on MsSqlHook implementation it uses the default timeout of the 
> driver  -> is no timeout.
> uses internally pymssql the default time is 0 -> is no timeout look at 
> http://pymssql.org/en/stable/ref/pymssql.html
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] eladkal opened a new pull request #4566: [AIRFLOW-3566] - Adding timeout and login_timeout to MsSqlHook

[GitBox]

eladkal opened a new pull request #4566: [AIRFLOW-3566] - Adding timeout and 
login_timeout to MsSqlHook
URL: https://github.com/apache/airflow/pull/4566
 
 
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3566) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3566
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   Adding timeout and login_timeout properties.
   timeout  - property of the query (query timeout in seconds).
   login_timeout -  property of connection (timeout for connection) allowing it 
to be set from Connection and overwrite in the hook.
   
   Default values (preserve backwards comparability):
   timeout default value is 0 (no timeout)
   login_timeout default value is 60
   
   
   ### Code Quality
   
   - [V] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mik-laj commented on a change in pull request #4524: [AIRFLOW-3707] Group subpackages by cloud providers

[GitBox]

mik-laj commented on a change in pull request #4524: [AIRFLOW-3707] Group 
subpackages by cloud providers
URL: https://github.com/apache/airflow/pull/4524#discussion_r249497315
 
 

 ##
 File path: setup.py
 ##
 @@ -211,27 +217,44 @@ def write_version(filename=os.path.join(*['airflow',
 qds = ['qds-sdk>=1.10.4']
 rabbitmq = ['librabbitmq>=1.6.1']
 redis = ['redis>=2.10.5,<3.0.0']
-s3 = ['boto3>=1.7.0, <1.8.0']
 salesforce = ['simple-salesforce>=0.72']
 samba = ['pysmbclient>=0.1.3']
 segment = ['analytics-python>=1.2.9']
 sendgrid = ['sendgrid>=5.2.0']
 slack = ['slackclient>=1.0.0']
 mongo = ['pymongo>=3.6.0']
-snowflake = ['snowflake-connector-python>=1.5.2',
- 'snowflake-sqlalchemy>=1.1.0']
-ssh = ['paramiko>=2.1.1', 'pysftp>=0.2.9', 'sshtunnel>=0.1.4,<0.2']
+snowflake = [
+'snowflake-connector-python>=1.5.2',
+'snowflake-sqlalchemy>=1.1.0',
+]
+ssh = [
+'paramiko>=2.1.1',
+'pysftp>=0.2.9',
+'sshtunnel>=0.1.4,<0.2',
+]
 statsd = ['statsd>=3.0.1, <4.0']
 vertica = ['vertica-python>=0.5.1']
 webhdfs = ['hdfs[dataframe,avro,kerberos]>=2.0.4']
 winrm = ['pywinrm==0.2.2']
 zendesk = ['zdesk']
 
-all_dbs = postgres + mysql + hive + mssql + hdfs + vertica + cloudant + druid 
+ pinot \
-+ cassandra + mongo
+all_dbs = (
+postgres +
+mysql +
+hive +
+mssql +
+hdfs +
+vertica +
+cloudant +
+druid +
+pinot +
+cassandra +
 
 Review comment:
   I will introduce the proposed changes today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4292: [AIRFLOW-2508] Handle non string types in Operators templatized fields

[GitBox]

codecov-io edited a comment on issue #4292: [AIRFLOW-2508] Handle non string 
types in Operators templatized fields
URL: https://github.com/apache/airflow/pull/4292#issuecomment-455914575
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=h1) 
Report
   > Merging 
[#4292](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/39dab6f4019f8107814104257274f3c7e78efeaa?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4292/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4292  +/-   ##
   ==
   - Coverage   74.11%   74.11%   -0.01% 
   ==
 Files 421  421  
 Lines   2768127657  -24 
   ==
   - Hits2051520497  -18 
   + Misses   7166 7160   -6
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.3% <100%> (-0.08%)` | :arrow_down: |
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `63.29% <0%> (-0.74%)` | :arrow_down: |
   | 
[airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=)
 | `92.1% <0%> (-0.21%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5)
 | `79.74% <0%> (-0.09%)` | :arrow_down: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4292/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `65.07% <0%> (+0.43%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=footer). 
Last update 
[39dab6f...798c79e](https://codecov.io/gh/apache/airflow/pull/4292?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4524: [AIRFLOW-3707] Group subpackages by cloud providers

[GitBox]

ashb commented on a change in pull request #4524: [AIRFLOW-3707] Group 
subpackages by cloud providers
URL: https://github.com/apache/airflow/pull/4524#discussion_r249492132
 
 

 ##
 File path: setup.py
 ##
 @@ -211,27 +217,44 @@ def write_version(filename=os.path.join(*['airflow',
 qds = ['qds-sdk>=1.10.4']
 rabbitmq = ['librabbitmq>=1.6.1']
 redis = ['redis>=2.10.5,<3.0.0']
-s3 = ['boto3>=1.7.0, <1.8.0']
 salesforce = ['simple-salesforce>=0.72']
 samba = ['pysmbclient>=0.1.3']
 segment = ['analytics-python>=1.2.9']
 sendgrid = ['sendgrid>=5.2.0']
 slack = ['slackclient>=1.0.0']
 mongo = ['pymongo>=3.6.0']
-snowflake = ['snowflake-connector-python>=1.5.2',
- 'snowflake-sqlalchemy>=1.1.0']
-ssh = ['paramiko>=2.1.1', 'pysftp>=0.2.9', 'sshtunnel>=0.1.4,<0.2']
+snowflake = [
+'snowflake-connector-python>=1.5.2',
+'snowflake-sqlalchemy>=1.1.0',
+]
+ssh = [
+'paramiko>=2.1.1',
+'pysftp>=0.2.9',
+'sshtunnel>=0.1.4,<0.2',
+]
 statsd = ['statsd>=3.0.1, <4.0']
 vertica = ['vertica-python>=0.5.1']
 webhdfs = ['hdfs[dataframe,avro,kerberos]>=2.0.4']
 winrm = ['pywinrm==0.2.2']
 zendesk = ['zdesk']
 
-all_dbs = postgres + mysql + hive + mssql + hdfs + vertica + cloudant + druid 
+ pinot \
-+ cassandra + mongo
+all_dbs = (
+postgres +
+mysql +
+hive +
+mssql +
+hdfs +
+vertica +
+cloudant +
+druid +
+pinot +
+cassandra +
 
 Review comment:
   I am in favour of the changes, but my personal preference is to keep 
white-space only/formatting only changes to their own commit (or even PR with 
AIRFLOW-XXX).
   
   If you are comfortable with git and pulling these changes out please do, 
otherwise in this case they can stay here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747996#comment-16747996
 ] 

ASF subversion and git services commented on AIRFLOW-1191:
--

Commit 39dab6f4019f8107814104257274f3c7e78efeaa in airflow's branch 
refs/heads/master from Holden Karau
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=39dab6f ]

[AIRFLOW-1191] Simplify override of spark submit command. (#4360)

Adds a spark_binary param to the spark submit operator to
allow folks to more easily configure the operator to use a
different binary, as is needed for some distros of the
Hadoop ecosystem which ship multiple version of Spark.

> Contrib Spark Submit hook should permit override of spark-submit cmd
> 
>
> Key: AIRFLOW-1191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks
>Affects Versions: 1.8.1
> Environment: Cloudera based Spark parcel
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
>Priority: Major
> Fix For: 1.10.0
>
>
> Using Cloudera based cluster with spark 2 parcel that rename spark-submit to 
> spark2-submit
> It should be possible to change the spark submit cmd without specifying a env 
> var



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-1191.
---
Resolution: Fixed
  Assignee: holdenk  (was: Vianney FOUCAULT)

> Contrib Spark Submit hook should permit override of spark-submit cmd
> 
>
> Key: AIRFLOW-1191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks
>Affects Versions: 1.8.1
> Environment: Cloudera based Spark parcel
>Reporter: Vianney FOUCAULT
>Assignee: holdenk
>Priority: Major
> Fix For: 2.0.0
>
>
> Using Cloudera based cluster with spark 2 parcel that rename spark-submit to 
> spark2-submit
> It should be possible to change the spark submit cmd without specifying a env 
> var



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-1191:
--
Fix Version/s: (was: 1.10.0)
   2.0.0

> Contrib Spark Submit hook should permit override of spark-submit cmd
> 
>
> Key: AIRFLOW-1191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks
>Affects Versions: 1.8.1
> Environment: Cloudera based Spark parcel
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
>Priority: Major
> Fix For: 2.0.0
>
>
> Using Cloudera based cluster with spark 2 parcel that rename spark-submit to 
> spark2-submit
> It should be possible to change the spark submit cmd without specifying a env 
> var



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747995#comment-16747995
 ] 

ASF GitHub Bot commented on AIRFLOW-1191:
-

Fokko commented on pull request #4360: [AIRFLOW-1191] Simplify override of 
spark submit command
URL: https://github.com/apache/airflow/pull/4360
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Contrib Spark Submit hook should permit override of spark-submit cmd
> 
>
> Key: AIRFLOW-1191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks
>Affects Versions: 1.8.1
> Environment: Cloudera based Spark parcel
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
>Priority: Major
> Fix For: 1.10.0
>
>
> Using Cloudera based cluster with spark 2 parcel that rename spark-submit to 
> spark2-submit
> It should be possible to change the spark submit cmd without specifying a env 
> var



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Fokko merged pull request #4360: [AIRFLOW-1191] Simplify override of spark submit command

[GitBox]

Fokko merged pull request #4360: [AIRFLOW-1191] Simplify override of spark 
submit command
URL: https://github.com/apache/airflow/pull/4360
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marengaz commented on issue #2753: [AIRFLOW-1782] Allow HttpSensor to support xcom

[GitBox]

marengaz commented on issue #2753: [AIRFLOW-1782] Allow HttpSensor to support 
xcom
URL: https://github.com/apache/airflow/pull/2753#issuecomment-456088301
 
 
   yes, this pr should be closed. #2753 implemented a global flag with this 
function on BaseOperator


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3743) Unify multiple ways of specifying and accessing AIRFLOW_HOME

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-3743:
--

 Summary: Unify multiple ways of specifying and accessing 
AIRFLOW_HOME
 Key: AIRFLOW-3743
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3743
 Project: Apache Airflow
  Issue Type: Improvement
  Components: configuration
Reporter: Ash Berlin-Taylor
Assignee: Ash Berlin-Taylor
 Fix For: 1.10.3


We currently have two (or three, depending on how you look at it) ways of 
getting AIRFLOW_HOME

{{airflow.configuration.AIRFLOW_HOME}}
{{airflow.settings.AIRFLOW_HOME}}
{{airflow.configuration.get('core', 'airflow_home')}}

used throughout the codebase. This has led to a few odd bugs 
(https://github.com/puckel/docker-airflow/issues/225 for example) if the config 
file and environment are out-of-sync.

Since the default path of the config file is dependent on the AIRFLOW_HOME 
environment variable I propose we remove the airflow_home from the config file, 
and unify all access in the code base to go via {{settings.AIRFLOW_HOME}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3742) airflow.configuration doesn't respect fallback kwarg

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-3742:
--

 Summary: airflow.configuration doesn't respect fallback kwarg
 Key: AIRFLOW-3742
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3742
 Project: Apache Airflow
  Issue Type: Improvement
  Components: configuration
Reporter: Ash Berlin-Taylor
Assignee: Ash Berlin-Taylor
 Fix For: 1.10.3


The ConfigParser class we base our config parser on can optionally accept a 
fallback argument:

{code:python}
conf.get('sect', 'key', fallback='default')
{code}

but our sub-class doesn't support this. It would be nice if we did. Then we can 
turn

{code:python}
plugins_folder = configuration.conf.get('core', 'plugins_folder')
if not plugins_folder:
plugins_folder = settings.AIRFLOW_HOME + '/plugins'
{code}

in to this:

{code:python}
plugins_folder = configuration.conf.get('core', 'plugins_folder', 
fallback=settings.AIRFLOW_HOME + '/plugins')
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)