date:20180808

codecov-io edited a comment on issue #3718: [AIRFLOW-2872] Implement 'Ad Hoc 
Query' for RBAC and Refine QueryView()
URL: 
https://github.com/apache/incubator-airflow/pull/3718#issuecomment-411331261
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=h1)
 Report
   > Merging 
[#3718](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **decrease** coverage by `0.2%`.
   > The diff coverage is `25%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3718/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3718  +/-   ##
   ==
   - Coverage   77.64%   77.43%   -0.21% 
   ==
 Files 204  204  
 Lines   1580115859  +58 
   ==
   + Hits1226812281  +13 
   - Misses   3533 3578  +45
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3718/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.88% <100%> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3718/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=)
 | `97.82% <100%> (+0.04%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3718/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.2% <17.5%> (-1.66%)` | :arrow_down: |
   | 
[airflow/www\_rbac/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3718/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy91dGlscy5weQ==)
 | `62.42% <29.41%> (-3.8%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=footer).
 Last update 
[8b04e20...6b13086](https://codecov.io/gh/apache/incubator-airflow/pull/3718?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2861) Need index on log table

2018-08-08 Thread Vardan Gupta (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vardan Gupta resolved AIRFLOW-2861.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

Change has been merged to Master.

> Need index on log table
> ---
>
> Key: AIRFLOW-2861
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2861
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: database
>Affects Versions: 1.10.0
>Reporter: Vardan Gupta
>Assignee: Vardan Gupta
>Priority: Major
> Fix For: 2.0.0
>
>
> Delete dag functionality is added in v1-10-stable, whose implementation 
> during the metadata cleanup 
> [part|https://github.com/apache/incubator-airflow/blob/dc78b9196723ca6724185231ccd6f5bbe8edcaf3/airflow/api/common/experimental/delete_dag.py#L48],
>  look for classes which has attribute named as dag_id and then formulate the 
> query on matching model and then delete from metadata, we've few numbers 
> where we've observed slowness especially in log table because it doesn't have 
> any single or multiple-column index. Creating an index would boost the 
> performance though insertion will be a bit slower. Since deletion will be a 
> sync call, would be good idea to create index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG commented on issue #3718: [AIRFLOW-2872] Implement 'Ad Hoc Query' for RBAC and Refine QueryView()

XD-DENG commented on issue #3718: [AIRFLOW-2872] Implement 'Ad Hoc Query' for 
RBAC and Refine QueryView()
URL: 
https://github.com/apache/incubator-airflow/pull/3718#issuecomment-411634875
 
 
   Hi @bolkedebruin , as suggested, I have implemented the **Ad Hoc Query** 
view for RBAC under `/www_rbac`. It's mainly based on the current 
implementation in `/www`. 
   
   I have updated the subject and contents of this PR accordingly as well.
   
   **Screenshot - Nav Bar**
   https://user-images.githubusercontent.com/11539188/43878430-ac307542-9bd1-11e8-988e-114daaff4b10.png;>
   
   **Screenshot - *Ad Hoc Query* view**
   https://user-images.githubusercontent.com/11539188/43878434-b2321a0e-9bd1-11e8-911d-9a841fc1ed7a.png;>
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-2872) Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView()

2018-08-08 Thread Xiaodong DENG (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-2872:
---
Description: 
To implement "Ad Hoc Query" in for RBAC in /www_rbac, based on the existing 
implementation in /www.

In addition, refine the existing QueryView():
 # The ".csv" button in *Ad Hoc Query* view is responding with a plain text 
file, rather than a CSV file (even though users can manually change the 
extension).
 # Argument 'has_data' passed to the template is not used by the template 
'airflow/query.html'.
 # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced 
before assignment'
 # 'result = df.to_html()' should only be invoked when user doesn NOT choose 
'.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the 
result it returns will not be used if user askes for CSV downloading instead of 
a html page.

  was:
# The ".csv" button in *Ad Hoc Query* view is responding with a plain text 
file, rather than a CSV file (even though users can manually change the 
extension).
 # Argument 'has_data' passed to the template is not used by the template 
'airflow/query.html'.
 # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced 
before assignment'
 # 'result = df.to_html()' should only be invoked when user doesn NOT choose 
'.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the 
result it returns will not be used if user askes for CSV downloading instead of 
a html page.

Summary: Implement "Ad Hoc Query" in /www_rbac, and refine existing 
QueryView()  (was: Minor bugs in "Ad Hoc Query" view, and refinement)

> Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView()
> --
>
> Key: AIRFLOW-2872
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2872
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> To implement "Ad Hoc Query" in for RBAC in /www_rbac, based on the existing 
> implementation in /www.
> In addition, refine the existing QueryView():
>  # The ".csv" button in *Ad Hoc Query* view is responding with a plain text 
> file, rather than a CSV file (even though users can manually change the 
> extension).
>  # Argument 'has_data' passed to the template is not used by the template 
> 'airflow/query.html'.
>  # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced 
> before assignment'
>  # 'result = df.to_html()' should only be invoked when user doesn NOT choose 
> '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the 
> result it returns will not be used if user askes for CSV downloading instead 
> of a html page.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on a change in pull request #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

feng-tao commented on a change in pull request #3648: [AIRFLOW-2786] Fix 
editing Variable with empty key crashing
URL: https://github.com/apache/incubator-airflow/pull/3648#discussion_r208799225
 
 

 ##
 File path: tests/www_rbac/test_views.py
 ##
 @@ -172,6 +172,26 @@ def test_xss_prevention(self):
 self.assertNotIn("",
  resp.data.decode("utf-8"))
 
+def test_import_variables(self):
+import mock
 
 Review comment:
   nit: put the import mock among others lib import.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-2686) Default Variables not base on default_timezone

2018-08-08 Thread Eric Luo (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Luo closed AIRFLOW-2686.
-
Resolution: Fixed

> Default Variables not base on default_timezone
> --
>
> Key: AIRFLOW-2686
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2686
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Eric Luo
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2686) Default Variables not base on default_timezone



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574189#comment-16574189
 ] 

ASF GitHub Bot commented on AIRFLOW-2686:
-

lxneng closed pull request #3554: [AIRFLOW-2686] Fix Default Variables not base 
on default_timezone
URL: https://github.com/apache/incubator-airflow/pull/3554
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/models.py b/airflow/models.py
index 260c0ba5a2..2fe229a685 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1809,14 +1809,15 @@ def get_template_context(self, session=None):
 tables = None
 if 'tables' in task.params:
 tables = task.params['tables']
+# convert to default timezone
+execution_date_tz = settings.TIMEZONE.convert(self.execution_date)
+ds = execution_date_tz.strftime('%Y-%m-%d')
+ts = execution_date_tz.isoformat()
+yesterday_ds = (execution_date_tz - timedelta(1)).strftime('%Y-%m-%d')
+tomorrow_ds = (execution_date_tz + timedelta(1)).strftime('%Y-%m-%d')
 
-ds = self.execution_date.strftime('%Y-%m-%d')
-ts = self.execution_date.isoformat()
-yesterday_ds = (self.execution_date - 
timedelta(1)).strftime('%Y-%m-%d')
-tomorrow_ds = (self.execution_date + timedelta(1)).strftime('%Y-%m-%d')
-
-prev_execution_date = task.dag.previous_schedule(self.execution_date)
-next_execution_date = task.dag.following_schedule(self.execution_date)
+prev_execution_date = task.dag.previous_schedule(execution_date_tz)
+next_execution_date = task.dag.following_schedule(execution_date_tz)
 
 next_ds = None
 if next_execution_date:
@@ -1903,7 +1904,7 @@ def __repr__(self):
 'end_date': ds,
 'dag_run': dag_run,
 'run_id': run_id,
-'execution_date': self.execution_date,
+'execution_date': execution_date_tz,
 'prev_execution_date': prev_execution_date,
 'next_execution_date': next_execution_date,
 'latest_date': ds,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Default Variables not base on default_timezone
> --
>
> Key: AIRFLOW-2686
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2686
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Eric Luo
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] lxneng commented on issue #3554: [AIRFLOW-2686] Fix Default Variables not base on default_timezone

lxneng commented on issue #3554: [AIRFLOW-2686] Fix Default Variables not base 
on default_timezone
URL: 
https://github.com/apache/incubator-airflow/pull/3554#issuecomment-411614674
 
 
   Agree with @Fokko , That would be mush better.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lxneng closed pull request #3554: [AIRFLOW-2686] Fix Default Variables not base on default_timezone

lxneng closed pull request #3554: [AIRFLOW-2686] Fix Default Variables not base 
on default_timezone
URL: https://github.com/apache/incubator-airflow/pull/3554
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/models.py b/airflow/models.py
index 260c0ba5a2..2fe229a685 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1809,14 +1809,15 @@ def get_template_context(self, session=None):
 tables = None
 if 'tables' in task.params:
 tables = task.params['tables']
+# convert to default timezone
+execution_date_tz = settings.TIMEZONE.convert(self.execution_date)
+ds = execution_date_tz.strftime('%Y-%m-%d')
+ts = execution_date_tz.isoformat()
+yesterday_ds = (execution_date_tz - timedelta(1)).strftime('%Y-%m-%d')
+tomorrow_ds = (execution_date_tz + timedelta(1)).strftime('%Y-%m-%d')
 
-ds = self.execution_date.strftime('%Y-%m-%d')
-ts = self.execution_date.isoformat()
-yesterday_ds = (self.execution_date - 
timedelta(1)).strftime('%Y-%m-%d')
-tomorrow_ds = (self.execution_date + timedelta(1)).strftime('%Y-%m-%d')
-
-prev_execution_date = task.dag.previous_schedule(self.execution_date)
-next_execution_date = task.dag.following_schedule(self.execution_date)
+prev_execution_date = task.dag.previous_schedule(execution_date_tz)
+next_execution_date = task.dag.following_schedule(execution_date_tz)
 
 next_ds = None
 if next_execution_date:
@@ -1903,7 +1904,7 @@ def __repr__(self):
 'end_date': ds,
 'dag_run': dag_run,
 'run_id': run_id,
-'execution_date': self.execution_date,
+'execution_date': execution_date_tz,
 'prev_execution_date': prev_execution_date,
 'next_execution_date': next_execution_date,
 'latest_date': ds,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3722: [AIRFLOW-2759] Add changes to extract proxy details at the base hook …

codecov-io edited a comment on issue #3722: [AIRFLOW-2759] Add changes to 
extract proxy details at the base hook …
URL: 
https://github.com/apache/incubator-airflow/pull/3722#issuecomment-411556404
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=h1)
 Report
   > Merging 
[#3722](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **decrease** coverage by `59.92%`.
   > The diff coverage is `15.38%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3722/graphs/tree.svg?token=WdLKlKHOAU=pr=150=650)](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#3722   +/-   ##
   ===
   - Coverage   77.64%   17.71%   -59.93% 
   ===
 Files 204  204   
 Lines   1580115826   +25 
   ===
   - Hits12268 2803 -9465 
   - Misses   353313023 +9490
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5)
 | `26.13% <15.38%> (-45.21%)` | :arrow_down: |
   | 
[airflow/hooks/base\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9iYXNlX2hvb2sucHk=)
 | `40.62% <15.38%> (-51.54%)` | :arrow_down: |
   | 
[airflow/hooks/slack\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zbGFja19ob29rLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/test\_utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvdGVzdF91dGlscy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/sensors/time\_delta\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3RpbWVfZGVsdGFfc2Vuc29yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/email\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZW1haWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/sensors/time\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3RpbWVfc2Vuc29yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/json.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9qc29uLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [163 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=footer).
 Last update 
[8b04e20...9742956](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] r39132 edited a comment on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12

r39132 edited a comment on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12
URL: 
https://github.com/apache/incubator-airflow/pull/3723#issuecomment-411582062
 
 
   @Fokko 
   I ran setup.py install on a branch based on this PR and I am getting 
exceptions when starting the scheduler.
   Here's the version of tenacity that was installed:
   ```
   sianand@LM-SJN-21002367:~/Projects/airflow_incubator $ pip freeze | grep 
tenacity
   tenacity==4.12.0
   ```
   When I run the scheduler, I see:
   ```
   
   [2018-08-08 16:06:35,130] {models.py:368} ERROR - Failed to import: 
/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/example_dags/example_http_operator.py
   Traceback (most recent call last):
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/models.py",
 line 365, in process_file
   m = imp.load_source(mod_name, filepath)
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py",
 line 172, in load_source
   module = _load(spec)
 File "", line 684, in _load
 File "", line 665, in _load_unlocked
 File "", line 678, in exec_module
 File "", line 219, in 
_call_with_frames_removed
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/example_dags/example_http_operator.py",
 line 27, in 
   from airflow.operators.http_operator import SimpleHttpOperator
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/operators/http_operator.py",
 line 21, in 
   from airflow.hooks.http_hook import HttpHook
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/hooks/http_hook.py",
 line 23, in 
   import tenacity
 File 
"/usr/local/lib/python3.6/site-packages/tenacity-4.12.0-py3.6.egg/tenacity/__init__.py",
 line 21, in 
   import asyncio
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/__init__.py",
 line 21, in 
   from .base_events import *
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
 line 17, in 
   import concurrent.futures
 File 
"/usr/local/lib/python3.6/site-packages/concurrent/futures/__init__.py", line 
8, in 
   from concurrent.futures._base import (FIRST_COMPLETED,
 File "/usr/local/lib/python3.6/site-packages/concurrent/futures/_base.py", 
line 381
   raise exception_type, self._exception, self._traceback
   ^
   SyntaxError: invalid syntax
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] r39132 commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12

r39132 commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12
URL: 
https://github.com/apache/incubator-airflow/pull/3723#issuecomment-411582062
 
 
   @Fokko 
   I ran setup.py install on a branch based on this PR and I am getting 
exceptions when starting the scheduler:
   ```
   sianand@LM-SJN-21002367:~/Projects/airflow_incubator $ pip freeze | grep 
tenacity
   tenacity==4.12.0
   
   [2018-08-08 16:06:35,130] {models.py:368} ERROR - Failed to import: 
/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/example_dags/example_http_operator.py
   Traceback (most recent call last):
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/models.py",
 line 365, in process_file
   m = imp.load_source(mod_name, filepath)
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py",
 line 172, in load_source
   module = _load(spec)
 File "", line 684, in _load
 File "", line 665, in _load_unlocked
 File "", line 678, in exec_module
 File "", line 219, in 
_call_with_frames_removed
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/example_dags/example_http_operator.py",
 line 27, in 
   from airflow.operators.http_operator import SimpleHttpOperator
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/operators/http_operator.py",
 line 21, in 
   from airflow.hooks.http_hook import HttpHook
 File 
"/usr/local/lib/python3.6/site-packages/apache_airflow-2.0.0.dev0+incubating-py3.6.egg/airflow/hooks/http_hook.py",
 line 23, in 
   import tenacity
 File 
"/usr/local/lib/python3.6/site-packages/tenacity-4.12.0-py3.6.egg/tenacity/__init__.py",
 line 21, in 
   import asyncio
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/__init__.py",
 line 21, in 
   from .base_events import *
 File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
 line 17, in 
   import concurrent.futures
 File 
"/usr/local/lib/python3.6/site-packages/concurrent/futures/__init__.py", line 
8, in 
   from concurrent.futures._base import (FIRST_COMPLETED,
 File "/usr/local/lib/python3.6/site-packages/concurrent/futures/_base.py", 
line 381
   raise exception_type, self._exception, self._traceback
   ^
   SyntaxError: invalid syntax
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12

codecov-io commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12
URL: 
https://github.com/apache/incubator-airflow/pull/3723#issuecomment-411565604
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=h1)
 Report
   > Merging 
[#3723](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3723/graphs/tree.svg?token=WdLKlKHOAU=pr=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3723   +/-   ##
   ===
 Coverage   77.64%   77.64%   
   ===
 Files 204  204   
 Lines   1580115801   
   ===
 Hits1226812268   
 Misses   3533 3533
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=footer).
 Last update 
[8b04e20...271ea66](https://codecov.io/gh/apache/incubator-airflow/pull/3723?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3658: [AIRFLOW-2524] Add Amazon SageMaker Training

codecov-io edited a comment on issue #3658: [AIRFLOW-2524] Add Amazon SageMaker 
Training
URL: 
https://github.com/apache/incubator-airflow/pull/3658#issuecomment-408564225
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=h1)
 Report
   > Merging 
[#3658](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/096ba9ecd961cdaebd062599f408571ffb21165a?src=pr=desc)
 will **increase** coverage by `0.52%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3658/graphs/tree.svg?height=150=WdLKlKHOAU=650=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3658  +/-   ##
   ==
   + Coverage   77.11%   77.63%   +0.52% 
   ==
 Files 206  204   -2 
 Lines   1577215801  +29 
   ==
   + Hits1216212267 +105 
   + Misses   3610 3534  -76
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5)
 | `66.92% <0%> (-1.08%)` | :arrow_down: |
   | 
[airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5)
 | `87.67% <0%> (-1.07%)` | :arrow_down: |
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `99.01% <0%> (-0.99%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/hive\_stats\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV9zdGF0c19vcGVyYXRvci5weQ==)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[airflow/sensors/hdfs\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2hkZnNfc2Vuc29yLnB5)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.88% <0%> (ø)` | :arrow_up: |
   | 
[airflow/hooks/presto\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wcmVzdG9faG9vay5weQ==)
 | `39.13% <0%> (ø)` | :arrow_up: |
   | 
[airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==)
 | `80.43% <0%> (ø)` | :arrow_up: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `64.35% <0%> (ø)` | :arrow_up: |
   | ... and [15 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3658/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=footer).
 Last update 
[096ba9e...2ef4f6f](https://codecov.io/gh/apache/incubator-airflow/pull/3658?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2876) Bump version of Tenacity



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573904#comment-16573904
 ] 

ASF GitHub Bot commented on AIRFLOW-2876:
-

Fokko opened a new pull request #3723: [AIRFLOW-2876] Update Tenacity to 4.12
URL: https://github.com/apache/incubator-airflow/pull/3723
 
 
   Tenacity 4.8 is not python 3.7 compatible because it contains reserved 
keywords in the code:
   ```
   [2018-08-08 21:21:22,016] {models.py:366} ERROR - Failed to import: 
/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py
   Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 363, 
in process_file
   m = imp.load_source(mod_name, filepath)
 File "/usr/local/lib/python3.7/imp.py", line 172, in load_source
   module = _load(spec)
 File "", line 696, in _load
 File "", line 677, in _load_unlocked
 File "", line 728, in exec_module
 File "", line 219, in 
_call_with_frames_removed
 File 
"/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py",
 line 27, in 
   from airflow.operators.http_operator import SimpleHttpOperator
 File 
"/usr/local/lib/python3.7/site-packages/airflow/operators/http_operator.py", 
line 21, in 
   from airflow.hooks.http_hook import HttpHook
 File "/usr/local/lib/python3.7/site-packages/airflow/hooks/http_hook.py", 
line 23, in 
   import tenacity
 File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 
352
   from tenacity.async import AsyncRetrying
   ```
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2876
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-2876\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Bump version of Tenacity
> 
>
> Key: AIRFLOW-2876
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2876
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>Priority: Major
>
> Since 4.8.0 is not Python 3.7 compatible, we want to bump the version to 
> 4.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with 
empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-411556954
 
 
   The tests are passing now, however instead of checking the POST response 
webpage, it checks the variables in the database.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Fokko opened a new pull request #3723: [AIRFLOW-2876] Update Tenacity to 4.12

Fokko opened a new pull request #3723: [AIRFLOW-2876] Update Tenacity to 4.12
URL: https://github.com/apache/incubator-airflow/pull/3723
 
 
   Tenacity 4.8 is not python 3.7 compatible because it contains reserved 
keywords in the code:
   ```
   [2018-08-08 21:21:22,016] {models.py:366} ERROR - Failed to import: 
/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py
   Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 363, 
in process_file
   m = imp.load_source(mod_name, filepath)
 File "/usr/local/lib/python3.7/imp.py", line 172, in load_source
   module = _load(spec)
 File "", line 696, in _load
 File "", line 677, in _load_unlocked
 File "", line 728, in exec_module
 File "", line 219, in 
_call_with_frames_removed
 File 
"/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py",
 line 27, in 
   from airflow.operators.http_operator import SimpleHttpOperator
 File 
"/usr/local/lib/python3.7/site-packages/airflow/operators/http_operator.py", 
line 21, in 
   from airflow.hooks.http_hook import HttpHook
 File "/usr/local/lib/python3.7/site-packages/airflow/hooks/http_hook.py", 
line 23, in 
   import tenacity
 File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 
352
   from tenacity.async import AsyncRetrying
   ```
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2876
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-2876\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3722: [AIRFLOW-2759] Add changes to extract proxy details at the base hook …

codecov-io edited a comment on issue #3722: [AIRFLOW-2759] Add changes to 
extract proxy details at the base hook …
URL: 
https://github.com/apache/incubator-airflow/pull/3722#issuecomment-411556404
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=h1)
 Report
   > Merging 
[#3722](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **decrease** coverage by `0.14%`.
   > The diff coverage is `46.15%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3722/graphs/tree.svg?token=WdLKlKHOAU=pr=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3722  +/-   ##
   ==
   - Coverage   77.64%   77.49%   -0.15% 
   ==
 Files 204  204  
 Lines   1580115826  +25 
   ==
   - Hits1226812264   -4 
   - Misses   3533 3562  +29
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/base\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9iYXNlX2hvb2sucHk=)
 | `76.56% <15.38%> (-15.6%)` | :arrow_down: |
   | 
[airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5)
 | `71.59% <76.92%> (+0.24%)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `71.42% <0%> (-10%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `88.6% <0%> (-0.85%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.49% <0%> (-0.27%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.76% <0%> (-0.13%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.78% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=footer).
 Last update 
[8b04e20...5aaf8ea](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3722: [AIRFLOW-2759] Add changes to extract proxy details at the base hook …

codecov-io commented on issue #3722: [AIRFLOW-2759] Add changes to extract 
proxy details at the base hook …
URL: 
https://github.com/apache/incubator-airflow/pull/3722#issuecomment-411556404
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=h1)
 Report
   > Merging 
[#3722](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **decrease** coverage by `0.14%`.
   > The diff coverage is `46.15%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3722/graphs/tree.svg?token=WdLKlKHOAU=pr=150=650)](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3722  +/-   ##
   ==
   - Coverage   77.64%   77.49%   -0.15% 
   ==
 Files 204  204  
 Lines   1580115826  +25 
   ==
   - Hits1226812264   -4 
   - Misses   3533 3562  +29
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/base\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9iYXNlX2hvb2sucHk=)
 | `76.56% <15.38%> (-15.6%)` | :arrow_down: |
   | 
[airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5)
 | `71.59% <76.92%> (+0.24%)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `71.42% <0%> (-10%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `88.6% <0%> (-0.85%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.49% <0%> (-0.27%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.76% <0%> (-0.13%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3722/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.78% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=footer).
 Last update 
[8b04e20...5aaf8ea](https://codecov.io/gh/apache/incubator-airflow/pull/3722?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2876) Bump version of Tenacity

2018-08-08 Thread Fokko Driesprong (JIRA)

Fokko Driesprong created AIRFLOW-2876:
-

 Summary: Bump version of Tenacity
 Key: AIRFLOW-2876
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2876
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Fokko Driesprong


Since 4.8.0 is not Python 3.7 compatible, we want to bump the version to 4.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable 
with empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-408263467
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=h1)
 Report
   > Merging 
[#3648](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3648/graphs/tree.svg?width=650=150=WdLKlKHOAU=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3648  +/-   ##
   ==
   + Coverage   77.64%   77.64%   +<.01% 
   ==
 Files 204  204  
 Lines   1580115825  +24 
   ==
   + Hits1226812287  +19 
   - Misses   3533 3538   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy91dGlscy5weQ==)
 | `67.1% <100%> (+0.88%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.04% <100%> (+0.15%)` | :arrow_up: |
   | 
[airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=)
 | `88.81% <100%> (+0.28%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.72% <60%> (-0.14%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.78% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=footer).
 Last update 
[8b04e20...e8669f1](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tedmiston commented on a change in pull request #3703: [AIRFLOW-2857] Fix broken RTD env

tedmiston commented on a change in pull request #3703: [AIRFLOW-2857] Fix 
broken RTD env 
URL: https://github.com/apache/incubator-airflow/pull/3703#discussion_r208734436
 
 

 ##
 File path: setup.py
 ##
 @@ -161,6 +164,7 @@ def write_version(filename=os.path.join(*['airflow',
 databricks = ['requests>=2.5.1, <3']
 datadog = ['datadog>=0.14.0']
 doc = [
+'mock',
 
 Review comment:
   @Fokko What Kaxil found is consistent with what I saw as well.  There's a 
bit more info on it in (1) in my comment here 
https://github.com/apache/incubator-airflow/pull/3703#issuecomment-410836135. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable 
with empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-408263467
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=h1)
 Report
   > Merging 
[#3648](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3648/graphs/tree.svg?token=WdLKlKHOAU=150=pr=650)](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3648  +/-   ##
   ==
   + Coverage   77.64%   77.64%   +<.01% 
   ==
 Files 204  204  
 Lines   1580115825  +24 
   ==
   + Hits1226812287  +19 
   - Misses   3533 3538   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy91dGlscy5weQ==)
 | `67.1% <100%> (+0.88%)` | :arrow_up: |
   | 
[airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=)
 | `88.81% <100%> (+0.28%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.04% <100%> (+0.15%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.72% <60%> (-0.14%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.78% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=footer).
 Last update 
[8b04e20...e8669f1](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

codecov-io edited a comment on issue #3648: [AIRFLOW-2786] Fix editing Variable 
with empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-408263467
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=h1)
 Report
   > Merging 
[#3648](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8b04e20709ebeb41aeefc0c5e3f12d35108ea504?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3648/graphs/tree.svg?token=WdLKlKHOAU=pr=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3648  +/-   ##
   ==
   + Coverage   77.64%   77.64%   +<.01% 
   ==
 Files 204  204  
 Lines   1580115825  +24 
   ==
   + Hits1226812287  +19 
   - Misses   3533 3538   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy91dGlscy5weQ==)
 | `67.1% <100%> (+0.88%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.04% <100%> (+0.15%)` | :arrow_up: |
   | 
[airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=)
 | `88.81% <100%> (+0.28%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `72.72% <60%> (-0.14%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3648/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.78% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=footer).
 Last update 
[8b04e20...e8669f1](https://codecov.io/gh/apache/incubator-airflow/pull/3648?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2759) Simplify proxy server based access to external platforms like Google cloud



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573840#comment-16573840
 ] 

ASF GitHub Bot commented on AIRFLOW-2759:
-

amohan34 opened a new pull request #3722: [AIRFLOW-2759] Add changes to extract 
proxy details at the base hook …
URL: https://github.com/apache/incubator-airflow/pull/3722
 
 
   …level
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-2759\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2759
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-2759\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   To test, run the following commands within tests/contrib/hooks
   
   nosetests test_gcp_api_base_hook_proxy.py --with-coverage 
--cover-package=airflow.contrib.hooks.gcp_api_base_hook
   
   
   ### Commits
   
   - [x†] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Simplify proxy server based access to external platforms like Google cloud 
> ---
>
> Key: AIRFLOW-2759
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2759
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Aishwarya Mohan
>Assignee: Aishwarya Mohan
>Priority: Major
>  Labels: hooks, proxy
>
> Several companies adopt a Proxy Server based approach in order to provide an 
> additional layer of security while communicating with external platforms to 
> establish legitimacy of caller and calle. A potential use case would be 
> writing logs from Airflow to a cloud storage platform like google cloud via 
> an intermediary proxy server.
> In the current scenario the proxy details need to be hardcoded and passed to 
> the HTTP client library(httplib2) in the GoogleCloudBaseHook class 
> (gcp_api_base_hook.py). It would be convenient if the proxy details (for 
> example, host and port) can be extracted from the airflow configuration file 
> as opposed to hardcoding the details at hook level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] amohan34 opened a new pull request #3722: [AIRFLOW-2759] Add changes to extract proxy details at the base hook …

amohan34 opened a new pull request #3722: [AIRFLOW-2759] Add changes to extract 
proxy details at the base hook …
URL: https://github.com/apache/incubator-airflow/pull/3722
 
 
   …level
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-2759\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2759
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-2759\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   To test, run the following commands within tests/contrib/hooks
   
   nosetests test_gcp_api_base_hook_proxy.py --with-coverage 
--cover-package=airflow.contrib.hooks.gcp_api_base_hook
   
   
   ### Commits
   
   - [x†] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bolkedebruin commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

bolkedebruin commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour 
of hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-411509935
 
 
   Of course! But the ball needs to get rolling too 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Fokko commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

Fokko commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-411506325
 
 
   @bolkedebruin My preference would be to get rid of the deprecation warnings 
and target it for Apache Airflow 2.0


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

2018-08-08 Thread ASF subversion and git services (JIRA)

feng-tao commented on a change in pull request #3714: [AIRFLOW-2867] Refactor 
code to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208683154
 
 

 ##
 File path: airflow/contrib/operators/mongo_to_s3.py
 ##
 @@ -105,7 +106,8 @@ def _stringify(self, iterable, joinable='\n'):
 [json.dumps(doc, default=json_util.default) for doc in iterable]
 )
 
-def transform(self, docs):
+@staticmethod
 
 Review comment:
   +1, I would also prefer static methods above instance methods


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573627#comment-16573627
 ] 

ASF subversion and git services commented on AIRFLOW-2856:
--

Commit 55200b48bbc0bf86db72c2b7eab01109bb98102f in incubator-airflow's branch 
refs/heads/v1-10-stable from [~ajc]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=55200b4 ]

[AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701)


(cherry picked from commit 8687ab9271b7b93473584a720f225f20fa9a7aa4)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 3670d49a4c5da4769be1df5b4f8ef3504b74a184)
Signed-off-by: Bolke de Bruin 


> Update Docker Env Setup Docs to Account for verify_gpl_dependency()
> ---
>
> Key: AIRFLOW-2856
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2856
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Andy Cooper
>Assignee: Andy Cooper
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2869) Remove smart quote from default config

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573623#comment-16573623
 ] 

ASF subversion and git services commented on AIRFLOW-2869:
--

Commit 26f6faed690846c963f90fe06458b3711d841e25 in incubator-airflow's branch 
refs/heads/v1-10-stable from [~wdhorton]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=26f6fae ]

[AIRFLOW-2869] Remove smart quote from default config

Closes #3716 from wdhorton/remove-smart-quote-
from-cfg

(cherry picked from commit 67e2bb96cdc5ea37226d11332362d3bd3778cea0)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 700f5f088dbead866170c9a3fe7e021e86ab30bb)
Signed-off-by: Bolke de Bruin 


> Remove smart quote from default config
> --
>
> Key: AIRFLOW-2869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2869
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Siddharth Anand
>Assignee: Siddharth Anand
>Priority: Trivial
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2859) DateTimes returned from the database are not converted to UTC

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573625#comment-16573625
 ] 

ASF subversion and git services commented on AIRFLOW-2859:
--

Commit a6d5ee9ce57bcdd22dde3fdf00e3b1cd26274ede in incubator-airflow's branch 
refs/heads/v1-10-stable from bolkedebruin
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=a6d5ee9 ]

[AIRFLOW-2859] Implement own UtcDateTime (#3708)

The different UtcDateTime implementations all have issues.
Either they replace tzinfo directly without converting
or they do not convert to UTC at all.

We also ensure all mysql connections are in UTC
in order to keep sanity, as mysql will ignore the
timezone of a field when inserting/updating.

(cherry picked from commit 6fd4e6055e36e9867923b0b402363fcd8c30e297)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 8fc8c7ae5483c002f5264b087b26a20fd8ae7b67)
Signed-off-by: Bolke de Bruin 


> DateTimes returned from the database are not converted to UTC
> -
>
> Key: AIRFLOW-2859
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2859
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.0
>
>
> This is due to the fact that sqlalchemy-utcdatetime does not convert to UTC 
> when the database returns datetimes with tzinfo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573619#comment-16573619
 ] 

ASF subversion and git services commented on AIRFLOW-2856:
--

Commit 3670d49a4c5da4769be1df5b4f8ef3504b74a184 in incubator-airflow's branch 
refs/heads/v1-10-test from [~ajc]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=3670d49 ]

[AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701)


(cherry picked from commit 8687ab9271b7b93473584a720f225f20fa9a7aa4)
Signed-off-by: Bolke de Bruin 


> Update Docker Env Setup Docs to Account for verify_gpl_dependency()
> ---
>
> Key: AIRFLOW-2856
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2856
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Andy Cooper
>Assignee: Andy Cooper
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573618#comment-16573618
 ] 

ASF subversion and git services commented on AIRFLOW-2870:
--

Commit 95aa49a71dcc69d2e9a8e32b69a2a61cacec2b1b in incubator-airflow's branch 
refs/heads/v1-10-test from bolkedebruin
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=95aa49a ]

[AIRFLOW-2870] Use abstract TaskInstance for migration (#3720)

If we use the full model for migration it can have columns
added that are not available yet in the database. Using
an abstraction ensures only the columns that are required
for data migration are present.

(cherry picked from commit 546f1cdb5208ba8e1cf3bde36bbdbb639fa20b22)
Signed-off-by: Bolke de Bruin 


> Migrations fail when upgrading from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance
> 
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> 
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", 
> line 1182, in _execute_context
> context)
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", 
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance 
> from the current code version, which has changes to the task_instance table 
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an 
> executor_config column that does not exist as of when 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because 
> the migration branches on table existence/non-existence at a point that will 
> hide the issue from new installs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training

troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add 
Amazon SageMaker Training
URL: https://github.com/apache/incubator-airflow/pull/3658#discussion_r208672024
 
 

 ##
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##
 @@ -0,0 +1,239 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import copy
+import time
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+from airflow.hooks.S3_hook import S3Hook
+
+
+class SageMakerHook(AwsHook):
+"""
+Interact with Amazon SageMaker.
+sagemaker_conn_id is required for using
+the config stored in db for training/tuning
+"""
+
+def __init__(self,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ region_name=None,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+super(SageMakerHook, self).__init__(*args, **kwargs)
+self.sagemaker_conn_id = sagemaker_conn_id
+self.use_db_config = use_db_config
+self.region_name = region_name
+self.check_interval = check_interval
+self.max_ingestion_time = max_ingestion_time
+self.conn = self.get_conn()
+
+def check_for_url(self, s3url):
+"""
+check if the s3url exists
+:param s3url: S3 url
+:type s3url:str
+:return: bool
+"""
+bucket, key = S3Hook.parse_s3_url(s3url)
+s3hook = S3Hook(aws_conn_id=self.aws_conn_id)
+if not s3hook.check_for_bucket(bucket_name=bucket):
+raise AirflowException(
+"The input S3 Bucket {} does not exist ".format(bucket))
+if not s3hook.check_for_key(key=key, bucket_name=bucket):
+raise AirflowException("The input S3 Key {} does not exist in the 
Bucket"
+   .format(s3url, bucket))
+return True
+
+def check_valid_training_input(self, training_config):
+"""
+Run checks before a training starts
+:param config: training_config
+:type config: dict
+:return: None
+"""
+for channel in training_config['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_valid_tuning_input(self, tuning_config):
+"""
+Run checks before a tuning job starts
+:param config: tuning_config
+:type config: dict
+:return: None
+"""
+for channel in 
tuning_config['TrainingJobDefinition']['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_status(self, non_terminal_states,
+ failed_state, key,
+ describe_function, *args):
+"""
+:param non_terminal_states: the set of non_terminal states
+:type non_terminal_states: dict
+:param failed_state: the set of failed states
+:type failed_state: dict
+:param key: the key of the response dict
+that points to the state
+:type key: string
+:param describe_function: the function used to retrieve the status
+:type describe_function: python callable
+:param args: the arguments for the function
+:return: None
+"""
+sec = 0
+running = True
+
+while running:
+
+sec = sec + self.check_interval
+
+if self.max_ingestion_time and sec > self.max_ingestion_time:
+# ensure that the job gets killed if the max ingestion time is 
exceeded
+raise AirflowException("SageMaker job took more than "
+   "%s seconds", self.max_ingestion_time)
+
+time.sleep(self.check_interval)
+try:
+status = describe_function(*args)[key]
+self.log.info("Job still running for %s seconds... "
+  "current status is %s" % (sec, status))
+

[GitHub] troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training

troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add 
Amazon SageMaker Training
URL: https://github.com/apache/incubator-airflow/pull/3658#discussion_r208671964
 
 

 ##
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##
 @@ -0,0 +1,239 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import copy
+import time
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+from airflow.hooks.S3_hook import S3Hook
+
+
+class SageMakerHook(AwsHook):
+"""
+Interact with Amazon SageMaker.
+sagemaker_conn_id is required for using
+the config stored in db for training/tuning
+"""
+
+def __init__(self,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ region_name=None,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+super(SageMakerHook, self).__init__(*args, **kwargs)
+self.sagemaker_conn_id = sagemaker_conn_id
+self.use_db_config = use_db_config
+self.region_name = region_name
+self.check_interval = check_interval
+self.max_ingestion_time = max_ingestion_time
+self.conn = self.get_conn()
+
+def check_for_url(self, s3url):
+"""
+check if the s3url exists
+:param s3url: S3 url
+:type s3url:str
+:return: bool
+"""
+bucket, key = S3Hook.parse_s3_url(s3url)
+s3hook = S3Hook(aws_conn_id=self.aws_conn_id)
+if not s3hook.check_for_bucket(bucket_name=bucket):
+raise AirflowException(
+"The input S3 Bucket {} does not exist ".format(bucket))
+if not s3hook.check_for_key(key=key, bucket_name=bucket):
+raise AirflowException("The input S3 Key {} does not exist in the 
Bucket"
+   .format(s3url, bucket))
+return True
+
+def check_valid_training_input(self, training_config):
+"""
+Run checks before a training starts
+:param config: training_config
+:type config: dict
+:return: None
+"""
+for channel in training_config['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_valid_tuning_input(self, tuning_config):
+"""
+Run checks before a tuning job starts
+:param config: tuning_config
+:type config: dict
+:return: None
+"""
+for channel in 
tuning_config['TrainingJobDefinition']['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_status(self, non_terminal_states,
+ failed_state, key,
+ describe_function, *args):
+"""
+:param non_terminal_states: the set of non_terminal states
+:type non_terminal_states: dict
+:param failed_state: the set of failed states
+:type failed_state: dict
+:param key: the key of the response dict
+that points to the state
+:type key: string
+:param describe_function: the function used to retrieve the status
+:type describe_function: python callable
+:param args: the arguments for the function
+:return: None
+"""
+sec = 0
+running = True
+
+while running:
+
+sec = sec + self.check_interval
+
+if self.max_ingestion_time and sec > self.max_ingestion_time:
+# ensure that the job gets killed if the max ingestion time is 
exceeded
+raise AirflowException("SageMaker job took more than "
+   "%s seconds", self.max_ingestion_time)
+
+time.sleep(self.check_interval)
+try:
+status = describe_function(*args)[key]
+self.log.info("Job still running for %s seconds... "
+  "current status is %s" % (sec, status))
+

[GitHub] troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training

troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add 
Amazon SageMaker Training
URL: https://github.com/apache/incubator-airflow/pull/3658#discussion_r208671902
 
 

 ##
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##
 @@ -0,0 +1,239 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import copy
+import time
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+from airflow.hooks.S3_hook import S3Hook
+
+
+class SageMakerHook(AwsHook):
+"""
+Interact with Amazon SageMaker.
+sagemaker_conn_id is required for using
+the config stored in db for training/tuning
+"""
+
+def __init__(self,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ region_name=None,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+super(SageMakerHook, self).__init__(*args, **kwargs)
+self.sagemaker_conn_id = sagemaker_conn_id
+self.use_db_config = use_db_config
+self.region_name = region_name
+self.check_interval = check_interval
+self.max_ingestion_time = max_ingestion_time
+self.conn = self.get_conn()
+
+def check_for_url(self, s3url):
+"""
+check if the s3url exists
+:param s3url: S3 url
+:type s3url:str
+:return: bool
+"""
+bucket, key = S3Hook.parse_s3_url(s3url)
+s3hook = S3Hook(aws_conn_id=self.aws_conn_id)
+if not s3hook.check_for_bucket(bucket_name=bucket):
+raise AirflowException(
+"The input S3 Bucket {} does not exist ".format(bucket))
+if not s3hook.check_for_key(key=key, bucket_name=bucket):
+raise AirflowException("The input S3 Key {} does not exist in the 
Bucket"
+   .format(s3url, bucket))
+return True
+
+def check_valid_training_input(self, training_config):
+"""
+Run checks before a training starts
+:param config: training_config
+:type config: dict
+:return: None
+"""
+for channel in training_config['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_valid_tuning_input(self, tuning_config):
+"""
+Run checks before a tuning job starts
+:param config: tuning_config
+:type config: dict
+:return: None
+"""
+for channel in 
tuning_config['TrainingJobDefinition']['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_status(self, non_terminal_states,
+ failed_state, key,
+ describe_function, *args):
+"""
+:param non_terminal_states: the set of non_terminal states
+:type non_terminal_states: dict
+:param failed_state: the set of failed states
+:type failed_state: dict
+:param key: the key of the response dict
+that points to the state
+:type key: string
+:param describe_function: the function used to retrieve the status
+:type describe_function: python callable
+:param args: the arguments for the function
+:return: None
+"""
+sec = 0
+running = True
+
+while running:
+
+sec = sec + self.check_interval
+
+if self.max_ingestion_time and sec > self.max_ingestion_time:
+# ensure that the job gets killed if the max ingestion time is 
exceeded
+raise AirflowException("SageMaker job took more than "
+   "%s seconds", self.max_ingestion_time)
+
+time.sleep(self.check_interval)
+try:
+status = describe_function(*args)[key]
+self.log.info("Job still running for %s seconds... "
+  "current status is %s" % (sec, status))
+

[jira] [Commented] (AIRFLOW-2875) Env variables should have percent signs escaped before writing to tmp config



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573560#comment-16573560
 ] 

ASF GitHub Bot commented on AIRFLOW-2875:
-

wdhorton opened a new pull request #3721: [AIRFLOW-2875] Escape env vars in tmp 
config
URL: https://github.com/apache/incubator-airflow/pull/3721
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   When writing tmp config files, escapes the env variables, since they are 
read by `ConfigParser` when the tmp config file is loaded when the task is 
being run.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Env variables should have percent signs escaped before writing to tmp config
> 
>
> Key: AIRFLOW-2875
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2875
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
> Environment: Ubuntu
> Airflow 1.10rc2
>Reporter: William Horton
>Priority: Major
>
> I encountered this when I was using an environment variable for 
> `AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and 
> communicate with the SQS queue, but when it received a task and began to run 
> it, I encountered an error with this trace:
> {code:java}
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring Traceback (most recent call last):
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring File "/opt/airflow/venv/bin/airflow", line 32, in 
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring args.func(args)
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring File 
> "/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py", 
> line 74, in wrapper
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring return f(*args, **kwargs)
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring File 
> "/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", 
> line 460, in run
> [2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring conf.set(section, option, value)
> [2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring File 
> "/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py",
>  line 1239, in set
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring super(ConfigParser, self).set(section, option, value)
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
> mirroring File 
> "/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py",
>  line 914, in set
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
>

[GitHub] wdhorton opened a new pull request #3721: [AIRFLOW-2875] Escape env vars in tmp config

wdhorton opened a new pull request #3721: [AIRFLOW-2875] Escape env vars in tmp 
config
URL: https://github.com/apache/incubator-airflow/pull/3721
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   When writing tmp config files, escapes the env variables, since they are 
read by `ConfigParser` when the tmp config file is loaded when the task is 
being run.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2826) Add hook for Google Cloud KMS

2018-08-08 Thread Jasper Kahn (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jasper Kahn resolved AIRFLOW-2826.
--
Resolution: Fixed

> Add hook for Google Cloud KMS
> -
>
> Key: AIRFLOW-2826
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2826
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Jasper Kahn
>Assignee: Jasper Kahn
>Priority: Minor
>  Labels: features
>
> Add a hook to support interacting with Google Cloud KMS. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

bolkedebruin edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in 
favour of hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-411484838
 
 
   Can you rebase and squash your commits? The we can (most likely :-) ) merge. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bolkedebruin edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

bolkedebruin edited a comment on issue #3560: [AIRFLOW-2697] Drop snakebite in 
favour of hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-411484838
 
 
   Can you revise and squash your commits? The we can (most likely :-) ) merge. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bolkedebruin commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

bolkedebruin commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour 
of hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-411484838
 
 
   Can you squash your commits? The we can (most likely :-) ) merge. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2861) Need index on log table

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573530#comment-16573530
 ] 

ASF GitHub Bot commented on AIRFLOW-2861:
-

bolkedebruin closed pull request #3709: [AIRFLOW-2861] Added index on log table
URL: https://github.com/apache/incubator-airflow/pull/3709
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py 
b/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py
new file mode 100644
index 00..3249a2e058
--- /dev/null
+++ b/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from alembic import op
+
+"""add idx_log_dag
+
+Revision ID: dd25f486b8ea
+Revises: 9635ae0956e7
+Create Date: 2018-08-07 06:41:41.028249
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'dd25f486b8ea'
+down_revision = '9635ae0956e7'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+op.create_index('idx_log_dag', 'log', ['dag_id'], unique=False)
+
+
+def downgrade():
+op.drop_index('idx_log_dag', table_name='log')
diff --git a/airflow/models.py b/airflow/models.py
index 288bd4c937..c79220805c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2098,6 +2098,10 @@ class Log(Base):
 owner = Column(String(500))
 extra = Column(Text)
 
+__table_args__ = (
+Index('idx_log_dag', dag_id),
+)
+
 def __init__(self, event, task_instance, owner=None, extra=None, **kwargs):
 self.dttm = timezone.utcnow()
 self.event = event


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Need index on log table
> ---
>
> Key: AIRFLOW-2861
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2861
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: database
>Affects Versions: 1.10.0
>Reporter: Vardan Gupta
>Assignee: Vardan Gupta
>Priority: Major
>
> Delete dag functionality is added in v1-10-stable, whose implementation 
> during the metadata cleanup 
> [part|https://github.com/apache/incubator-airflow/blob/dc78b9196723ca6724185231ccd6f5bbe8edcaf3/airflow/api/common/experimental/delete_dag.py#L48],
>  look for classes which has attribute named as dag_id and then formulate the 
> query on matching model and then delete from metadata, we've few numbers 
> where we've observed slowness especially in log table because it doesn't have 
> any single or multiple-column index. Creating an index would boost the 
> performance though insertion will be a bit slower. Since deletion will be a 
> sync call, would be good idea to create index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2861) Need index on log table



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573531#comment-16573531
 ] 

ASF subversion and git services commented on AIRFLOW-2861:
--

Commit 6f7fe74b9ff6c2abae2764988c152af8bcc8e199 in incubator-airflow's branch 
refs/heads/master from [~vardan]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6f7fe74 ]

[AIRFLOW-2861] Add index on log table (#3709)



> Need index on log table
> ---
>
> Key: AIRFLOW-2861
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2861
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: database
>Affects Versions: 1.10.0
>Reporter: Vardan Gupta
>Assignee: Vardan Gupta
>Priority: Major
>
> Delete dag functionality is added in v1-10-stable, whose implementation 
> during the metadata cleanup 
> [part|https://github.com/apache/incubator-airflow/blob/dc78b9196723ca6724185231ccd6f5bbe8edcaf3/airflow/api/common/experimental/delete_dag.py#L48],
>  look for classes which has attribute named as dag_id and then formulate the 
> query on matching model and then delete from metadata, we've few numbers 
> where we've observed slowness especially in log table because it doesn't have 
> any single or multiple-column index. Creating an index would boost the 
> performance though insertion will be a bit slower. Since deletion will be a 
> sync call, would be good idea to create index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin closed pull request #3709: [AIRFLOW-2861] Added index on log table

bolkedebruin closed pull request #3709: [AIRFLOW-2861] Added index on log table
URL: https://github.com/apache/incubator-airflow/pull/3709
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py 
b/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py
new file mode 100644
index 00..3249a2e058
--- /dev/null
+++ b/airflow/migrations/versions/dd25f486b8ea_add_idx_log_dag.py
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from alembic import op
+
+"""add idx_log_dag
+
+Revision ID: dd25f486b8ea
+Revises: 9635ae0956e7
+Create Date: 2018-08-07 06:41:41.028249
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'dd25f486b8ea'
+down_revision = '9635ae0956e7'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+op.create_index('idx_log_dag', 'log', ['dag_id'], unique=False)
+
+
+def downgrade():
+op.drop_index('idx_log_dag', table_name='log')
diff --git a/airflow/models.py b/airflow/models.py
index 288bd4c937..c79220805c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2098,6 +2098,10 @@ class Log(Base):
 owner = Column(String(500))
 extra = Column(Text)
 
+__table_args__ = (
+Index('idx_log_dag', dag_id),
+)
+
 def __init__(self, event, task_instance, owner=None, extra=None, **kwargs):
 self.dttm = timezone.utcnow()
 self.event = event


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-2875) Env variables should have percent signs escaped before writing to tmp config

2018-08-08 Thread William Horton (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

William Horton updated AIRFLOW-2875:

Description: 
I encountered this when I was using an environment variable for 
`AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and 
communicate with the SQS queue, but when it received a task and began to run 
it, I encountered an error with this trace:
{code:java}
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring Traceback (most recent call last):
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File "/opt/airflow/venv/bin/airflow", line 32, in 
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring args.func(args)
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py", 
line 74, in wrapper
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring return f(*args, **kwargs)
[2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 
460, in run
[2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring conf.set(section, option, value)
[2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py",
 line 1239, in set
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring super(ConfigParser, self).set(section, option, value)
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py",
 line 914, in set
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring value)
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py",
 line 392, in before_set
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring "position %d" % (value, tmp_value.find('%')))
[2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring ValueError: invalid interpolation syntax in 
{code}
The issue was that the broker url had a percent sign, and when the cli called 
`conf.set(section, option, value)`, it was throwing because it interpreted the 
percent as an interpolation.

To avoid this issue, I would propose that the environment variables be escaped 
when being written in `utils.configuration.tmp_configuration_copy`, so that 
when `conf.set` is called in `bin/cli`, it doesn't throw on these unescaped 
values.

  was:
I encountered this when I was using an environment variable for 
`AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and 
communicate with the SQS queue, but when it received a task and began to run 
it, I encountered an error with this trace:
```
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring Traceback (most recent call last):
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File "/opt/airflow/venv/bin/airflow", line 32, in 
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring args.func(args)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py", 
line 74, in wrapper
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring return f(*args, **kwargs)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 
460, in run
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring conf.set(section, option, value)
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
 line 1239, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring super(ConfigParser, self).set(section, option, value)
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
 line 914, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring value)
[2018-08-08 15:19:24,406]

[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573527#comment-16573527
 ] 

ASF GitHub Bot commented on AIRFLOW-2870:
-

bolkedebruin closed pull request #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: https://github.com/apache/incubator-airflow/pull/3720
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
 
b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
index b7213a3031..27a9f593b5 100644
--- 
a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
+++ 
b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#   http://www.apache.org/licenses/LICENSE-2.0
 #
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 
 """kubernetes_resource_checkpointing
 
diff --git 
a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
 
b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
index 4347bae92a..c489c05f7e 100644
--- 
a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
+++ 
b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#   http://www.apache.org/licenses/LICENSE-2.0
 #
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 
 """kubernetes_resource_checkpointing
 
diff --git 
a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
 
b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
index 6bc48f1105..5c921c6a98 100644
--- 
a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
+++ 
b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more

[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573528#comment-16573528
 ] 

ASF subversion and git services commented on AIRFLOW-2870:
--

Commit 546f1cdb5208ba8e1cf3bde36bbdbb639fa20b22 in incubator-airflow's branch 
refs/heads/master from bolkedebruin
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=546f1cd ]

[AIRFLOW-2870] Use abstract TaskInstance for migration (#3720)

If we use the full model for migration it can have columns
added that are not available yet in the database. Using
an abstraction ensures only the columns that are required
for data migration are present.

> Migrations fail when upgrading from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance
> 
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> 
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", 
> line 1182, in _execute_context
> context)
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", 
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance 
> from the current code version, which has changes to the task_instance table 
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an 
> executor_config column that does not exist as of when 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because 
> the migration branches on table existence/non-existence at a point that will 
> hide the issue from new installs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin closed pull request #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

bolkedebruin closed pull request #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: https://github.com/apache/incubator-airflow/pull/3720
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
 
b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
index b7213a3031..27a9f593b5 100644
--- 
a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
+++ 
b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#   http://www.apache.org/licenses/LICENSE-2.0
 #
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 
 """kubernetes_resource_checkpointing
 
diff --git 
a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
 
b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
index 4347bae92a..c489c05f7e 100644
--- 
a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
+++ 
b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#   http://www.apache.org/licenses/LICENSE-2.0
 #
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 
 """kubernetes_resource_checkpointing
 
diff --git 
a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
 
b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
index 6bc48f1105..5c921c6a98 100644
--- 
a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
+++ 
b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
@@ -1,16 +1,22 @@
 # flake8: noqa
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may

[jira] [Created] (AIRFLOW-2875) Env variables should have percent signs escaped before writing to tmp config

2018-08-08 Thread William Horton (JIRA)

William Horton created AIRFLOW-2875:
---

 Summary: Env variables should have percent signs escaped before 
writing to tmp config
 Key: AIRFLOW-2875
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2875
 Project: Apache Airflow
  Issue Type: Bug
  Components: configuration
 Environment: Ubuntu
Airflow 1.10rc2
Reporter: William Horton


I encountered this when I was using an environment variable for 
`AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and 
communicate with the SQS queue, but when it received a task and began to run 
it, I encountered an error with this trace:
```
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring Traceback (most recent call last):
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File "/opt/airflow/venv/bin/airflow", line 32, in 
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring args.func(args)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py", 
line 74, in wrapper
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring return f(*args, **kwargs)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 
460, in run
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring conf.set(section, option, value)
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
 line 1239, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring super(ConfigParser, self).set(section, option, value)
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
 line 914, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring value)
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring File 
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
 line 392, in before_set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring "position %d" % (value, tmp_value.find('%')))
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask 
mirroring ValueError: invalid interpolation syntax in 
```

The issue was that the broker url had a percent sign, and when the cli called 
`conf.set(section, option, value)`, it was throwing because it interpreted the 
percent as an interpolation.

To avoid this issue, I would propose that the environment variables be escaped 
when being written in `utils.configuration.tmp_configuration_copy`, so that 
when `conf.set` is called in `bin/cli`, it doesn't throw on these unescaped 
values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] gwax commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

gwax commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for 
migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411480640
 
 
   That works for me now.
   
   :shipit: !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

codecov-io edited a comment on issue #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411401924
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=h1)
 Report
   > Merging 
[#3720](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8687ab9271b7b93473584a720f225f20fa9a7aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3720/graphs/tree.svg?token=WdLKlKHOAU=pr=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3720   +/-   ##
   ===
 Coverage   77.63%   77.63%   
   ===
 Files 204  204   
 Lines   1580015800   
   ===
 Hits1226712267   
 Misses   3533 3533
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=footer).
 Last update 
[8687ab9...311bc91](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bolkedebruin commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

bolkedebruin commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance 
for migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411469023
 
 
   @gwax done. PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bolkedebruin commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

bolkedebruin commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance 
for migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411468352
 
 
   Ah let me fix that


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gwax commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

gwax commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for 
migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411461955
 
 
   Clever approach; thanks for taking this on @bolkedebruin 
   
   On attempting `upgradedb`, I run into the following error with this branch:
   
   ```
 File 
"/Users/georgelesliewaksman/.pyenv/versions/2.7.15/envs/temp2/lib/python2.7/site-packages/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py",
 line 98, in upgrade
   ti.max_tries = ti.try_number
   AttributeError: 'TaskInstance' object has no attribute 'try_number'
   ```
   
   Looks like we also need the "try_number" column on the migration version of 
the model:
   
   ```
   try_number = Column(Integer, default=0)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-2868) Mesos Executor should use executor_config to specify CPU, Memory and Docker image on the task level

2018-08-08 Thread Amir Shahatit (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-2868 started by Amir Shahatit.
--
> Mesos Executor should use executor_config to specify CPU, Memory and Docker 
> image on the task level
> ---
>
> Key: AIRFLOW-2868
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2868
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Affects Versions: 1.10, 1.10.1
>Reporter: Amir Shahatit
>Assignee: Amir Shahatit
>Priority: Major
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Executor_config was added as a part of 
> [AIRFLOW-1314|https://github.com/apache/incubator-airflow/commit/c0920efc012468681cff3d3c9cfe25c7381dc976].
>  This task extends the mesosExecutor to make use of specified executor 
> configs to pass on resource requirements (CPU/Memory) as well as docker 
> images on the task level. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] vardancse commented on issue #3709: [AIRFLOW-2861] Added index on log table

vardancse commented on issue #3709: [AIRFLOW-2861] Added index on log table
URL: 
https://github.com/apache/incubator-airflow/pull/3709#issuecomment-411409483
 
 
   @bolkedebruin @feng-tao @Fokko  Do you guys see any other concern?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

codecov-io edited a comment on issue #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411401924
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=h1)
 Report
   > Merging 
[#3720](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8687ab9271b7b93473584a720f225f20fa9a7aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3720/graphs/tree.svg?src=pr=WdLKlKHOAU=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3720   +/-   ##
   ===
 Coverage   77.63%   77.63%   
   ===
 Files 204  204   
 Lines   1580015800   
   ===
 Hits1226712267   
 Misses   3533 3533
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=footer).
 Last update 
[8687ab9...8c55dfa](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

codecov-io commented on issue #3720: [AIRFLOW-2870] Use abstract TaskInstance 
for migration
URL: 
https://github.com/apache/incubator-airflow/pull/3720#issuecomment-411401924
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=h1)
 Report
   > Merging 
[#3720](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8687ab9271b7b93473584a720f225f20fa9a7aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3720/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3720   +/-   ##
   ===
 Coverage   77.63%   77.63%   
   ===
 Files 204  204   
 Lines   1580015800   
   ===
 Hits1226712267   
 Misses   3533 3533
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=footer).
 Last update 
[8687ab9...8c55dfa](https://codecov.io/gh/apache/incubator-airflow/pull/3720?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573199#comment-16573199
 ] 

Kaxil Naik edited comment on AIRFLOW-2871 at 8/8/18 1:07 PM:
-

[~ashb] [~tedmiston] I have added a stable version which would also point to 
the stable release. So we now have the latest, stable and other old versions.

Check the image below:

!screenshot-3.png!

*Note*: We wouldn't have docs for <=1.8.2 as readthedocs confs were broken in 
that release.

I think this should suffice for now. But if you guys think we should add a 
banner as well that notifies if someone is viewing unstable (dev or old) 
release, we can use 
https://robpol86.github.io/sphinxcontrib-versioning/v2.1.1/banner.html and 
follow the thread here https://github.com/rtfd/readthedocs.org/issues/4327 . 
[~tedmiston] You can play with docs by setting readthedocs for forked repo like 
I have done at https://readthedocs.org/projects/airflow-fork-k1/. 

Stable Release Docs: https://airflow.apache.org/ | 
https://airflow.readthedocs.io/en/stable/
Dev Release (master branch docs): https://airflow.readthedocs.io/en/latest/

RTD will also now host all the versioned docs from 1.9.0:
- https://airflow.readthedocs.io/en/1.9.0/


was (Author: kaxilnaik):
[~ashb] [~tedmiston] I have added a stable version which would also point to 
the stable release. So we now have the latest, stable and other old versions.

Check the image below:

!screenshot-3.png!

*Note*: We wouldn't have docs for <=1.8.2 as readthedocs confs were broken in 
that release.

I think this should suffice for now. But if you guys think we should add a 
banner as well that notifies if someone is viewing unstable (dev or old) 
release, we can use 
https://robpol86.github.io/sphinxcontrib-versioning/v2.1.1/banner.html and 
follow the thread here https://github.com/rtfd/readthedocs.org/issues/4327 . 
[~tedmiston] You can play with docs by setting readthedocs for forked repo like 
I have done at https://readthedocs.org/projects/airflow-fork-k1/. 

Note that our main docs would still be at https://airflow.apache.org/

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
>

[jira] [Comment Edited] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573199#comment-16573199
 ] 

Kaxil Naik edited comment on AIRFLOW-2871 at 8/8/18 1:05 PM:
-

[~ashb] [~tedmiston] I have added a stable version which would also point to 
the stable release. So we now have the latest, stable and other old versions.

Check the image below:

!screenshot-3.png!

*Note*: We wouldn't have docs for <=1.8.2 as readthedocs confs were broken in 
that release.

I think this should suffice for now. But if you guys think we should add a 
banner as well that notifies if someone is viewing unstable (dev or old) 
release, we can use 
https://robpol86.github.io/sphinxcontrib-versioning/v2.1.1/banner.html and 
follow the thread here https://github.com/rtfd/readthedocs.org/issues/4327 . 
[~tedmiston] You can play with docs by setting readthedocs for forked repo like 
I have done at https://readthedocs.org/projects/airflow-fork-k1/. 

Note that our main docs would still be at https://airflow.apache.org/


was (Author: kaxilnaik):
[~ashb] [~tedmiston] I have added a stable version which would also point to 
the stable release. So we now have the latest, stable and other old versions.

Check the image below:

!screenshot-3.png!

*Note*: We wouldn't have docs for <=1.8.2 as readthedocs confs were broken in 
that release.

I think this should suffice for now. But if you guys think we should add a 
banner as well that notifies if someone is viewing unstable (dev or old) 
release, we can use 
https://robpol86.github.io/sphinxcontrib-versioning/v2.1.1/banner.html and 
follow the thread here https://github.com/rtfd/readthedocs.org/issues/4327 . 
[~tedmiston] You can play with docs by setting readthedocs for forked repo like 
I have done at https://readthedocs.org/projects/airflow-fork-k1/. 

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed

[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573199#comment-16573199
 ] 

Kaxil Naik commented on AIRFLOW-2871:
-

[~ashb] [~tedmiston] I have added a stable version which would also point to 
the stable release. So we now have the latest, stable and other old versions.

Check the image below:

!screenshot-3.png!

*Note*: We wouldn't have docs for <=1.8.2 as readthedocs confs were broken in 
that release.

I think this should suffice for now. But if you guys think we should add a 
banner as well that notifies if someone is viewing unstable (dev or old) 
release, we can use 
https://robpol86.github.io/sphinxcontrib-versioning/v2.1.1/banner.html and 
follow the thread here https://github.com/rtfd/readthedocs.org/issues/4327 . 
[~tedmiston] You can play with docs by setting readthedocs for forked repo like 
I have done at https://readthedocs.org/projects/airflow-fork-k1/. 

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using

[jira] [Updated] (AIRFLOW-2871) Harden and improve Read the Docs build environment



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-2871:

Attachment: screenshot-3.png

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe 
> this would require.  I am happy to take this issue if I can get that.  I have 
> experience as an admin of another project on RTD (simple-salesforce).
> 
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411388413
 
 
   @ashb, thanks! 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

codecov-io edited a comment on issue #3717: [AIRFLOW-1874] use_legacy_sql added 
to BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411257030
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=h1)
 Report
   > Merging 
[#3717](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/acca61c602e341da06ebee2eca3a26f4e7400238?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3717/graphs/tree.svg?width=650=150=WdLKlKHOAU=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3717   +/-   ##
   ===
 Coverage   77.63%   77.63%   
   ===
 Files 204  204   
 Lines   1580015800   
   ===
 Hits1226712267   
 Misses   3533 3533
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=footer).
 Last update 
[acca61c...ce9fe5c](https://codecov.io/gh/apache/incubator-airflow/pull/3717?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

2018-08-08 Thread ASF subversion and git services (JIRA)

ashb commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411381407
 
 
   I don't think that error log was a test failure - from the name of the file 
it is designed to be an error.
   
   The failure was one of the Kubernetes test envs timedout/hung: 
https://travis-ci.org/apache/incubator-airflow/jobs/413500015
   
   I'm retryuing that one failed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573089#comment-16573089
 ] 

ASF subversion and git services commented on AIRFLOW-2856:
--

Commit 8687ab9271b7b93473584a720f225f20fa9a7aa4 in incubator-airflow's branch 
refs/heads/master from [~ajc]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8687ab9 ]

[AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701)



> Update Docker Env Setup Docs to Account for verify_gpl_dependency()
> ---
>
> Key: AIRFLOW-2856
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2856
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Andy Cooper
>Assignee: Andy Cooper
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()

2018-08-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573088#comment-16573088
 ] 

ASF GitHub Bot commented on AIRFLOW-2856:
-

bolkedebruin closed pull request #3701: [AIRFLOW-2856] Pass in 
SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run
URL: https://github.com/apache/incubator-airflow/pull/3701
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 2cf8e0218e..5d3c83aa14 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -86,10 +86,7 @@ Go to your Airflow directory and start a new docker 
container. You can choose be
 
 ```
 # Start docker in your Airflow directory
-docker run -t -i -v `pwd`:/airflow/ python:2 bash
-
-# Go to the Airflow directory
-cd /airflow/
+docker run -t -i -v `pwd`:/airflow/ -w /airflow/ -e 
SLUGIFY_USES_TEXT_UNIDECODE=yes python:2 bash
 
 # Install Airflow with all the required dependencies,
 # including the devel which will provide the development tools


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update Docker Env Setup Docs to Account for verify_gpl_dependency()
> ---
>
> Key: AIRFLOW-2856
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2856
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Andy Cooper
>Assignee: Andy Cooper
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2874) Enable Flask App Builder theme support



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573086#comment-16573086
 ] 

ASF subversion and git services commented on AIRFLOW-2874:
--

Commit cbe7955d7e0dd352829f71e762026f4044a05a7d in incubator-airflow's branch 
refs/heads/master from [~verdan]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=cbe7955 ]

[AIRFLOW-2874] Enables FAB's theme support (#3719)



> Enable Flask App Builder theme support
> --
>
> Key: AIRFLOW-2874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2874
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Verdan Mahmood
>Assignee: Verdan Mahmood
>Priority: Major
>
> To customize the look and feel of Apache Airflow (an effort towards making 
> Airflow a whitelabel application), we should enable the support of FAB's 
> theme, which can be set in configuration. 
> Theme can be use in conjunction of existing `navbar_color` configuration or 
> can be used separately by simple unsetting the navbar_color config. 
>  
> http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2874) Enable Flask App Builder theme support



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573085#comment-16573085
 ] 

ASF GitHub Bot commented on AIRFLOW-2874:
-

bolkedebruin closed pull request #3719: [AIRFLOW-2874] Enables FAB's theme 
support
URL: https://github.com/apache/incubator-airflow/pull/3719
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/config_templates/default_webserver_config.py 
b/airflow/config_templates/default_webserver_config.py
index ba33d81dec..e61c7e1a45 100644
--- a/airflow/config_templates/default_webserver_config.py
+++ b/airflow/config_templates/default_webserver_config.py
@@ -89,3 +89,13 @@
 #{ 'name': 'AOL', 'url': 'http://openid.aol.com/' },
 #{ 'name': 'Flickr', 'url': 'http://www.flickr.com/' },
 #{ 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
+
+# 
+# Theme CONFIG
+# 
+# Flask App Builder comes up with a number of predefined themes
+# that you can use for Apache Airflow.
+# 
http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
+# Please make sure to remove "navbar_color" configuration from airflow.cfg
+# in order to fully utilize the theme. (or use that property in conjunction 
with theme)
+# APP_THEME = "bootstrap-theme.css"  # default bootstrap
diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index 6b0c4416a9..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -20,8 +20,14 @@
 
 {% block head_css %}
   {{ super() }}
-  {# styleBundle.css file contains the styles from main.css and 
bootstrap-theme.css #}
-  
+
+  
+
+  {% if not appbuilder.app_theme %}
+{# airflowDefaultTheme.css file contains the styles from local 
bootstrap-theme.css #}
+
+  {% endif %}
+
   
 {% endblock %}
 
diff --git a/airflow/www_rbac/webpack.config.js 
b/airflow/www_rbac/webpack.config.js
index 46df449a50..29b99f408f 100644
--- a/airflow/www_rbac/webpack.config.js
+++ b/airflow/www_rbac/webpack.config.js
@@ -38,10 +38,8 @@ const config = {
 clock: `${STATIC_DIR}/js/clock.js`,
 graph: `${STATIC_DIR}/js/graph.js`,
 ganttChartD3v2: `${STATIC_DIR}/js/gantt-chart-d3v2.js`,
-styleBundle: [
-  `${STATIC_DIR}/css/bootstrap-theme.css`,
-  `${STATIC_DIR}/css/main.css`,
-],
+main: `${STATIC_DIR}/css/main.css`,
+airflowDefaultTheme : `${STATIC_DIR}/css/bootstrap-theme.css`,
   },
   output: {
 path: BUILD_DIR,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable Flask App Builder theme support
> --
>
> Key: AIRFLOW-2874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2874
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Verdan Mahmood
>Assignee: Verdan Mahmood
>Priority: Major
>
> To customize the look and feel of Apache Airflow (an effort towards making 
> Airflow a whitelabel application), we should enable the support of FAB's 
> theme, which can be set in configuration. 
> Theme can be use in conjunction of existing `navbar_color` configuration or 
> can be used separately by simple unsetting the navbar_color config. 
>  
> http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin closed pull request #3719: [AIRFLOW-2874] Enables FAB's theme support

bolkedebruin closed pull request #3719: [AIRFLOW-2874] Enables FAB's theme 
support
URL: https://github.com/apache/incubator-airflow/pull/3719
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/config_templates/default_webserver_config.py 
b/airflow/config_templates/default_webserver_config.py
index ba33d81dec..e61c7e1a45 100644
--- a/airflow/config_templates/default_webserver_config.py
+++ b/airflow/config_templates/default_webserver_config.py
@@ -89,3 +89,13 @@
 #{ 'name': 'AOL', 'url': 'http://openid.aol.com/' },
 #{ 'name': 'Flickr', 'url': 'http://www.flickr.com/' },
 #{ 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
+
+# 
+# Theme CONFIG
+# 
+# Flask App Builder comes up with a number of predefined themes
+# that you can use for Apache Airflow.
+# 
http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
+# Please make sure to remove "navbar_color" configuration from airflow.cfg
+# in order to fully utilize the theme. (or use that property in conjunction 
with theme)
+# APP_THEME = "bootstrap-theme.css"  # default bootstrap
diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index 6b0c4416a9..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -20,8 +20,14 @@
 
 {% block head_css %}
   {{ super() }}
-  {# styleBundle.css file contains the styles from main.css and 
bootstrap-theme.css #}
-  
+
+  
+
+  {% if not appbuilder.app_theme %}
+{# airflowDefaultTheme.css file contains the styles from local 
bootstrap-theme.css #}
+
+  {% endif %}
+
   
 {% endblock %}
 
diff --git a/airflow/www_rbac/webpack.config.js 
b/airflow/www_rbac/webpack.config.js
index 46df449a50..29b99f408f 100644
--- a/airflow/www_rbac/webpack.config.js
+++ b/airflow/www_rbac/webpack.config.js
@@ -38,10 +38,8 @@ const config = {
 clock: `${STATIC_DIR}/js/clock.js`,
 graph: `${STATIC_DIR}/js/graph.js`,
 ganttChartD3v2: `${STATIC_DIR}/js/gantt-chart-d3v2.js`,
-styleBundle: [
-  `${STATIC_DIR}/css/bootstrap-theme.css`,
-  `${STATIC_DIR}/css/main.css`,
-],
+main: `${STATIC_DIR}/css/main.css`,
+airflowDefaultTheme : `${STATIC_DIR}/css/bootstrap-theme.css`,
   },
   output: {
 path: BUILD_DIR,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3675: [AIRFLOW-2834] fix build script for k8s docker

codecov-io commented on issue #3675: [AIRFLOW-2834] fix build script for k8s 
docker
URL: 
https://github.com/apache/incubator-airflow/pull/3675#issuecomment-411377800
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=h1)
 Report
   > Merging 
[#3675](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/acca61c602e341da06ebee2eca3a26f4e7400238?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3675/graphs/tree.svg?token=WdLKlKHOAU=pr=650=150)](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3675  +/-   ##
   ==
   + Coverage   77.63%   77.64%   +<.01% 
   ==
 Files 204  204  
 Lines   1580015800  
   ==
   + Hits1226712268   +1 
   + Misses   3533 3532   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/3675/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `89.87% <0%> (+0.42%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=footer).
 Last update 
[acca61c...f8fb4dd](https://codecov.io/gh/apache/incubator-airflow/pull/3675?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573041#comment-16573041
 ] 

ASF GitHub Bot commented on AIRFLOW-2870:
-

bolkedebruin opened a new pull request #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: https://github.com/apache/incubator-airflow/pull/3720
 
 
   
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2870
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   If we use the full model for migration it can have columns
   added that are not available yet in the database. Using
   an abstraction ensures only the columns that are required
   for data migration are present.
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Db migration
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   
   @ashb @gwax PTAL
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Migrations fail when upgrading from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance
> 
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> 
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", 
> line 1182, in _execute_context
> context)
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", 
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance 
> from the current code version, which has changes to the task_instance table 
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an 
> executor_config column that does not exist as of when 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because 
> the migration branches on table existence/non-existence at a point that will 
> hide the issue from new installs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin opened a new pull request #3720: [AIRFLOW-2870] Use abstract TaskInstance for migration

bolkedebruin opened a new pull request #3720: [AIRFLOW-2870] Use abstract 
TaskInstance for migration
URL: https://github.com/apache/incubator-airflow/pull/3720
 
 
   
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2870
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   If we use the full model for migration it can have columns
   added that are not available yet in the database. Using
   an abstraction ensures only the columns that are required
   for data migration are present.
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Db migration
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   
   @ashb @gwax PTAL
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support

codecov-io commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support
URL: 
https://github.com/apache/incubator-airflow/pull/3719#issuecomment-411367530
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=h1)
 Report
   > Merging 
[#3719](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/acca61c602e341da06ebee2eca3a26f4e7400238?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3719/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#3719   +/-   ##
   ===
 Coverage   77.63%   77.63%   
   ===
 Files 204  204   
 Lines   1580015800   
   ===
 Hits1226712267   
 Misses   3533 3533
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[...rflow/config\_templates/default\_webserver\_config.py](https://codecov.io/gh/apache/incubator-airflow/pull/3719/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfd2Vic2VydmVyX2NvbmZpZy5weQ==)
 | `0% <ø> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=footer).
 Last update 
[acca61c...df2831f](https://codecov.io/gh/apache/incubator-airflow/pull/3719?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411360602
 
 
   tests failed with 
   
   ERROR [airflow.models.DagBag] Failed to bag_dag: 
/home/travis/build/apache/incubator-airflow/tests/dags/test_zip_invalid_cron.zip
   
   seems, like tests in master broken


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2874) Enable Flask App Builder theme support



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572989#comment-16572989
 ] 

ASF GitHub Bot commented on AIRFLOW-2874:
-

verdan opened a new pull request #3719: [AIRFLOW-2874] Enables FAB's theme 
support
URL: https://github.com/apache/incubator-airflow/pull/3719
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
2874](https://issues.apache.org/jira/browse/AIRFLOW-2874) issues and references 
them in the PR title.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   In an effort to make Apache Airflow a white label application, we should 
enable the FAB's theme support. Users can then easily change the theme 
according to their needs. There already exists a configuration to set the 
Navbar Color, that can be used in conjunction with the theme option, or can 
simply be unset to fully utilize the theme. List of available themes are 
available here: 
https://github.com/dpgaspar/Flask-AppBuilder-Skeleton/blob/master/config.py#L88
   _Note: Some of the dark themes might not work properly because of the charts 
that we are using at the moment._
   Here are some of the screenshots from different themes:
   
   ![screen shot 2018-08-08 at 11 41 17 
am](https://user-images.githubusercontent.com/25360476/43831843-bb6705d4-9b05-11e8-85cc-a818fb48c77a.png)
   
   ![screen shot 2018-08-08 at 11 42 05 
am](https://user-images.githubusercontent.com/25360476/43831844-bb812356-9b05-11e8-8429-3b06c8cf1a7a.png)
   
   ![screen shot 2018-08-08 at 11 42 48 
am](https://user-images.githubusercontent.com/25360476/43831846-bb9b7ada-9b05-11e8-9396-b8c9a67d84fa.png)
   
   ![screen shot 2018-08-08 at 11 43 20 
am](https://user-images.githubusercontent.com/25360476/43831847-bbb150da-9b05-11e8-98e3-29f32d0b71e1.png)
   
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable Flask App Builder theme support
> --
>
> Key: AIRFLOW-2874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2874
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Verdan Mahmood
>Assignee: Verdan Mahmood
>Priority: Major
>
> To customize the look and feel of Apache Airflow (an effort towards making 
> Airflow a whitelabel application), we should enable the support of FAB's 
> theme, which can be set in configuration. 
> Theme can be use in conjunction of existing `navbar_color` configuration or 
> can be used separately by simple unsetting the navbar_color config. 
>  
> http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] verdan opened a new pull request #3719: [AIRFLOW-2874] Enables FAB's theme support

verdan opened a new pull request #3719: [AIRFLOW-2874] Enables FAB's theme
support
URL: https://github.com/apache/incubator-airflow/pull/3719

Make sure you have checked _all_ steps below.

### Jira

- [ ] My PR addresses the following [Airflow
2874](https://issues.apache.org/jira/browse/AIRFLOW-2874) issues and references
them in the PR title.

### Description

- [ ] Here are some details about my PR, including screenshots of any UI
changes:
In an effort to make Apache Airflow a white label application, we should
enable the FAB's theme support. Users can then easily change the theme
according to their needs. There already exists a configuration to set the
Navbar Color, that can be used in conjunction with the theme option, or can
simply be unset to fully utilize the theme. List of available themes are
available here:
https://github.com/dpgaspar/Flask-AppBuilder-Skeleton/blob/master/config.py#L88
_Note: Some of the dark themes might not work properly because of the charts
that we are using at the moment._
Here are some of the screenshots from different themes:

![screen shot 2018-08-08 at 11 41 17
am](https://user-images.githubusercontent.com/25360476/43831843-bb6705d4-9b05-11e8-85cc-a818fb48c77a.png)

![screen shot 2018-08-08 at 11 42 05
am](https://user-images.githubusercontent.com/25360476/43831844-bb812356-9b05-11e8-8429-3b06c8cf1a7a.png)

![screen shot 2018-08-08 at 11 42 48
am](https://user-images.githubusercontent.com/25360476/43831846-bb9b7ada-9b05-11e8-9396-b8c9a67d84fa.png)

![screen shot 2018-08-08 at 11 43 20
am](https://user-images.githubusercontent.com/25360476/43831847-bbb150da-9b05-11e8-98e3-29f32d0b71e1.png)

### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.

### Code Quality

- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-2874) Enable Flask App Builder theme support

2018-08-08 Thread Verdan Mahmood (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Verdan Mahmood updated AIRFLOW-2874:

Description: 
To customize the look and feel of Apache Airflow (an effort towards making 
Airflow a whitelabel application), we should enable the support of FAB's theme, 
which can be set in configuration. 

Theme can be use in conjunction of existing `navbar_color` configuration or can 
be used separately by simple unsetting the navbar_color config. 

 

http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes

  was:
To customize the look and feel of Apache Airflow, we should enable the support 
of FAB's theme, which can be set in configuration. 

Theme can be use in conjunction of existing `navbar_color` configuration or can 
be used separately by simple unsetting the navbar_color config. 


> Enable Flask App Builder theme support
> --
>
> Key: AIRFLOW-2874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2874
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Verdan Mahmood
>Assignee: Verdan Mahmood
>Priority: Major
>
> To customize the look and feel of Apache Airflow (an effort towards making 
> Airflow a whitelabel application), we should enable the support of FAB's 
> theme, which can be set in configuration. 
> Theme can be use in conjunction of existing `navbar_color` configuration or 
> can be used separately by simple unsetting the navbar_color config. 
>  
> http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2874) Enable Flask App Builder theme support

2018-08-08 Thread Verdan Mahmood (JIRA)

Verdan Mahmood created AIRFLOW-2874:
---

 Summary: Enable Flask App Builder theme support
 Key: AIRFLOW-2874
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2874
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Verdan Mahmood
Assignee: Verdan Mahmood


To customize the look and feel of Apache Airflow, we should enable the support 
of FAB's theme, which can be set in configuration. 

Theme can be use in conjunction of existing `navbar_color` configuration or can 
be used separately by simple unsetting the navbar_color config. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside edited a comment on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

xnuinside edited a comment on issue #3717: [AIRFLOW-1874] use_legacy_sql added 
to BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411349555
 
 
   @kaxil , yeah, sure, first time I have done it the way as you write, but 
then found close PR with the reason of  
https://issues.apache.org/jira/browse/AIRFLOW-559 ) was confused and made 
changes. Any way, done. And thanks for review!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

xnuinside commented on a change in pull request #3714: [AIRFLOW-2867] Refactor 
code to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208520636
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -238,6 +238,8 @@ def create_empty_table(self,
 
 :return:
 """
+if time_partitioning is None:
+time_partitioning = dict()
 
 Review comment:
   @kaxil , @ashb , thx!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

xnuinside commented on issue #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: 
https://github.com/apache/incubator-airflow/pull/3717#issuecomment-411349555
 
 
   @kaxil , yeah, sure, first time I have done it the way as you write, but 
then found close PR with the reason of  
https://issues.apache.org/jira/browse/AIRFLOW-559 ) was confused an made 
changes. Any way, done. And thanks for review!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

kaxil commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code 
to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208519487
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -238,6 +238,8 @@ def create_empty_table(self,
 
 :return:
 """
+if time_partitioning is None:
+time_partitioning = dict()
 
 Review comment:
   @xnuinside Hi, there is no specific reason for me to use `{}` compared to 
`dict()`. The point for this change as I mentioned earlier was to remove an 
empty dictionary from default arguments. 
   
   And as Ash pointed out `{}` dict literal is faster that dict constructor 
`dict()` but there is not huge difference as the dict builds up. Check out the 
below links for more detailed read:
   - 
https://doughellmann.com/blog/2012/11/12/the-performance-impact-of-using-dict-instead-of-in-cpython-2-7-2/
   - 
https://stackoverflow.com/questions/6610606/is-there-a-difference-between-using-a-dict-literal-and-a-dict-constructor


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

kaxil commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code 
to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208519487
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -238,6 +238,8 @@ def create_empty_table(self,
 
 :return:
 """
+if time_partitioning is None:
+time_partitioning = dict()
 
 Review comment:
   @xnuinside Hi, there is no specific reason for me to use `{}` compared to 
`dict()`. The point for this change as I mentioned earlier was to remove an 
empty list from default arguments. 
   
   And as Ash pointed out `{}` dict literal is faster that dict constructor 
`dict()` but there is not huge difference as the dict builds up. Check out the 
below links for more detailed read:
   - 
https://doughellmann.com/blog/2012/11/12/the-performance-impact-of-using-dict-instead-of-in-cpython-2-7-2/
   - 
https://stackoverflow.com/questions/6610606/is-there-a-difference-between-using-a-dict-literal-and-a-dict-constructor


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

ashb commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code 
to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208515928
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -238,6 +238,8 @@ def create_empty_table(self,
 
 :return:
 """
+if time_partitioning is None:
+time_partitioning = dict()
 
 Review comment:
   `{}` would probably be better in hindsight, but there's not much in it. On 
my laptop:
   
   ```
   In [5]: %timeit {}
   47.2 ns ± 2.3 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
   
   In [6]: %timeit dict()
   168 ns ± 1.77 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
   ```
   
   so yes dict() is 3 times as "slow" as `{}`, but neither is particularly slow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2873) Improvements to Quick Start flow



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572924#comment-16572924
 ] 

G. Geijteman commented on AIRFLOW-2873:
---

[~ashb]

Thank you, I could now run the installation phase without a problem.

I suppose the Quick start will be fixed when 1.10 is out of RC then.

 

One small thing, I do get a warning when I try:
{code:java}
airflow initdb{code}
Results in:
{code:java}
~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py:154: 
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead 
exc_info=1) 
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - 
values will not be stored encrypted. Traceback (most recent call last): File 
"~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", line 
147, in get_fernet from cryptography.fernet import Fernet, InvalidToken 
ModuleNotFoundError: No module named 'cryptography' 
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] Could not import 
KubernetesPodOperator: No module named 'kubernetes' WARNI 
[airflow.utils.log.logging_mixin.LoggingMixin] Install kubernetes dependencies 
with: pip install airflow['kubernetes']{code}
It's cool to see Kubernetes is supported, but it now seems a requirement for 
the database init. That seems a little silly. I suppose that's still work in 
progress on the master?

 

> Improvements to Quick Start flow
> 
>
> Key: AIRFLOW-2873
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2873
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Affects Versions: Airflow 1.9.0
>Reporter: G. Geijteman
>Priority: Major
>
> Thank you for developing Airflow!
> Having ran through the [Quick 
> Start|https://airflow.incubator.apache.org/start.html], i've come across two 
> issues that I would like to highlight:
> {code:java}
> bash-3.2$ cd ~/project/airflow/
> bash-3.2$ export AIRFLOW_HOME=~/project/airflow
> bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
> bash-3.2$ source venv/bin/activate
> (venv) bash-3.2$ pip install --upgrade pip
> (venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 
> 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 
> x86_64
> (venv) bash-3.2$ python -V
> Python 3.6.5
> (venv) bash-3.2$ pip -V
> pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
> 3.6)
> (venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
> Results in:
> {code:java}
> During handling of the above exception, another exception occurred:Traceback 
> (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 639, in set_extra
> fernet = get_fernet()
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 103, in get_fernet
> raise AirflowException('Failed to import Fernet, it may not be installed')
> airflow.exceptions.AirflowException: Failed to import Fernet, it may not be 
> installed
> [2018-08-08 10:49:01,121]{models.py:643}ERROR - Failed to load fernet while 
> encrypting value, using non-encrypted value.
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 101, in get_fernet
> from cryptography.fernet import Fernet
> ModuleNotFoundError: No module named 'cryptography'{code}
> This is solved by:
> {code:java}
> (venv) bash-3.2$ pip install cryptography{code}
> *Proposed fix:*
> _Include the `cryptography` package in the setup / package requirements_
>  
> Having fixed that, the following issue occurs when trying to:
> {code:java}
> (venv) bash-3.2$ airflow initdb{code}
> Exempt:
> {code:java}
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 639, in set_extra
> fernet = get_fernet()
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 107, in get_fernet
> raise AirflowException("Could not create Fernet object: {}".format(ve))
> airflow.exceptions.AirflowException: Could not create Fernet object: 
> Incorrect padding
> [2018-08-08 10:50:50,697]
> {models.py:643}
> ERROR - Failed to load fernet while encrypting value, using non-encrypted 
> value.
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 105, in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
> File 
> "~/project/airflow/venv/lib/python3.6/site-packages/cryptography/fernet.py", 
> line 34, in _init_
> key = base64.urlsafe_b64decode(key)
> File 
>

[GitHub] xnuinside commented on a change in pull request #3714: [AIRFLOW-2867] Refactor code to conform Python standards & guidelines

xnuinside commented on a change in pull request #3714: [AIRFLOW-2867] Refactor 
code to conform Python standards & guidelines
URL: https://github.com/apache/incubator-airflow/pull/3714#discussion_r208514928
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -238,6 +238,8 @@ def create_empty_table(self,
 
 :return:
 """
+if time_partitioning is None:
+time_partitioning = dict()
 
 Review comment:
   @kaxil  but why dict() instead of {}?  just want to understand for my self
as I know {} more efficient 
https://stackoverflow.com/questions/664118/whats-the-difference-between-dict-and.
 In official docs cannot see any recommendations to use dict() instead {} 
https://docs.python.org/3.6/library/stdtypes.html#dict 
   and in standard library used {} not dict() 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572921#comment-16572921
 ] 

Kaxil Naik commented on AIRFLOW-2871:
-

[~tedmiston] Few pointers:

- We are using VirtualEnv setup for RTD
- Based on the discussion above with Ash, I will also make some distinction 
between latest vs stable.
- I remember previously I had wiped out the cache. I will redo that.

If there are any issues for me to do anything list, I will add you to RTD.



> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe 
> this would require.  I am happy to take this issue if I can get that.  I have 
> experience as an admin of another project on RTD (simple-salesforce).
> 
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA

[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572916#comment-16572916
 ] 

Kaxil Naik commented on AIRFLOW-2871:
-

[~ashb] I will sort that out with a new PR and ping you once it is ready for 
review.

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe 
> this would require.  I am happy to take this issue if I can get that.  I have 
> experience as an admin of another project on RTD (simple-salesforce).
> 
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2873) Improvements to Quick Start flow

2018-08-08 Thread Ash Berlin-Taylor (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572914#comment-16572914
 ] 

Ash Berlin-Taylor commented on AIRFLOW-2873:


Thanks for reporting this. An easy quickstart procedure is definitely something 
we value!

The hard-dependency on cryptography should have been removed in 
master/1.10.0rc3 The padding issue might be as well.

Could you try again after installing this?

{code:bash}
AIRFLOW_GPL_UNIDECODE=yes pip install 
'https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz#egg=apache-airflow>=1.10'
{code}

> Improvements to Quick Start flow
> 
>
> Key: AIRFLOW-2873
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2873
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Affects Versions: Airflow 1.9.0
>Reporter: G. Geijteman
>Priority: Major
>
> Thank you for developing Airflow!
> Having ran through the [Quick 
> Start|https://airflow.incubator.apache.org/start.html], i've come across two 
> issues that I would like to highlight:
> {code:java}
> bash-3.2$ cd ~/project/airflow/
> bash-3.2$ export AIRFLOW_HOME=~/project/airflow
> bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
> bash-3.2$ source venv/bin/activate
> (venv) bash-3.2$ pip install --upgrade pip
> (venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 
> 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 
> x86_64
> (venv) bash-3.2$ python -V
> Python 3.6.5
> (venv) bash-3.2$ pip -V
> pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
> 3.6)
> (venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
> Results in:
> {code:java}
> During handling of the above exception, another exception occurred:Traceback 
> (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 639, in set_extra
> fernet = get_fernet()
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 103, in get_fernet
> raise AirflowException('Failed to import Fernet, it may not be installed')
> airflow.exceptions.AirflowException: Failed to import Fernet, it may not be 
> installed
> [2018-08-08 10:49:01,121]{models.py:643}ERROR - Failed to load fernet while 
> encrypting value, using non-encrypted value.
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 101, in get_fernet
> from cryptography.fernet import Fernet
> ModuleNotFoundError: No module named 'cryptography'{code}
> This is solved by:
> {code:java}
> (venv) bash-3.2$ pip install cryptography{code}
> *Proposed fix:*
> _Include the `cryptography` package in the setup / package requirements_
>  
> Having fixed that, the following issue occurs when trying to:
> {code:java}
> (venv) bash-3.2$ airflow initdb{code}
> Exempt:
> {code:java}
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 639, in set_extra
> fernet = get_fernet()
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 107, in get_fernet
> raise AirflowException("Could not create Fernet object: {}".format(ve))
> airflow.exceptions.AirflowException: Could not create Fernet object: 
> Incorrect padding
> [2018-08-08 10:50:50,697]
> {models.py:643}
> ERROR - Failed to load fernet while encrypting value, using non-encrypted 
> value.
> Traceback (most recent call last):
> File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
> line 105, in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
> File 
> "~/project/airflow/venv/lib/python3.6/site-packages/cryptography/fernet.py", 
> line 34, in _init_
> key = base64.urlsafe_b64decode(key)
> File 
> "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
>  line 133, in urlsafe_b64decode
> return b64decode(s)
> File 
> "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
>  line 87, in b64decode
> return binascii.a2b_base64(s)
> binascii.Error: Incorrect padding{code}
> Which after some googling leads to the conclusion that the 
> ~/project/airflow/airflow.cfg fernet_key field is not set to the correct 
> value.
> *Feature request:*
> _Have the setup automatically generate a valid fernet key for the user._
> The fact that this page exists: [https://bcb.github.io/airflow/fernet-key] 
> suggests this could easily be a part of the package.
> I understand that this project is in incubator phase, but I would say having 
> a quick start that

[jira] [Updated] (AIRFLOW-2873) Improvements to Quick Start flow



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

G. Geijteman updated AIRFLOW-2873:
--
Description: 
Thank you for developing Airflow!

Having ran through the [Quick 
Start|https://airflow.incubator.apache.org/start.html], i've come across two 
issues that I would like to highlight:
{code:java}
bash-3.2$ cd ~/project/airflow/
bash-3.2$ export AIRFLOW_HOME=~/project/airflow
bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$ pip install --upgrade pip
(venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 17.7.0: 
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
(venv) bash-3.2$ python -V
Python 3.6.5
(venv) bash-3.2$ pip -V
pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
3.6)
(venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
Results in:
{code:java}
During handling of the above exception, another exception occurred:Traceback 
(most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 103, in get_fernet
raise AirflowException('Failed to import Fernet, it may not be installed')
airflow.exceptions.AirflowException: Failed to import Fernet, it may not be 
installed
[2018-08-08 10:49:01,121]{models.py:643}ERROR - Failed to load fernet while 
encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 101, in get_fernet
from cryptography.fernet import Fernet
ModuleNotFoundError: No module named 'cryptography'{code}
This is solved by:
{code:java}
(venv) bash-3.2$ pip install cryptography{code}
*Proposed fix:*

_Include the `cryptography` package in the setup / package requirements_

 

Having fixed that, the following issue occurs when trying to:
{code:java}
(venv) bash-3.2$ airflow initdb{code}
Exempt:
{code:java}
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 107, in get_fernet
raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect 
padding
[2018-08-08 10:50:50,697]

{models.py:643}

ERROR - Failed to load fernet while encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 105, in get_fernet
return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
File 
"~/project/airflow/venv/lib/python3.6/site-packages/cryptography/fernet.py", 
line 34, in _init_
key = base64.urlsafe_b64decode(key)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 133, in urlsafe_b64decode
return b64decode(s)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding{code}
Which after some googling leads to the conclusion that the 
~/project/airflow/airflow.cfg fernet_key field is not set to the correct value.

*Feature request:*

_Have the setup automatically generate a valid fernet key for the user._

The fact that this page exists: [https://bcb.github.io/airflow/fernet-key] 
suggests this could easily be a part of the package.

I understand that this project is in incubator phase, but I would say having a 
quick start that is not working as-is will discourage users from trying out 
this project. Thank you for considering.

  was:
Thank you for developing Airflow!

Having ran through the Quick Start, i've come across two issues that I would 
like to highlight:
{code:java}
bash-3.2$ cd ~/project/airflow/
bash-3.2$ export AIRFLOW_HOME=~/project/airflow
bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$ pip install --upgrade pip
(venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 17.7.0: 
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
(venv) bash-3.2$ python -V
Python 3.6.5
(venv) bash-3.2$ pip -V
pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
3.6)
(venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
Results in:
{code:java}
During handling of the above exception, another exception occurred:Traceback 
(most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in

[jira] [Updated] (AIRFLOW-2873) Improvements to Quick Start flow



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

G. Geijteman updated AIRFLOW-2873:
--
Description: 
Thank you for developing Airflow!

Having ran through the Quick Start, i've come across two issues that I would 
like to highlight:
{code:java}
bash-3.2$ cd ~/project/airflow/
bash-3.2$ export AIRFLOW_HOME=~/project/airflow
bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$ pip install --upgrade pip
(venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 17.7.0: 
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
(venv) bash-3.2$ python -V
Python 3.6.5
(venv) bash-3.2$ pip -V
pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
3.6)
(venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
Results in:
{code:java}
During handling of the above exception, another exception occurred:Traceback 
(most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 103, in get_fernet
raise AirflowException('Failed to import Fernet, it may not be installed')
airflow.exceptions.AirflowException: Failed to import Fernet, it may not be 
installed
[2018-08-08 10:49:01,121]{models.py:643}ERROR - Failed to load fernet while 
encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 101, in get_fernet
from cryptography.fernet import Fernet
ModuleNotFoundError: No module named 'cryptography'{code}
This is solved by:
{code:java}
(venv) bash-3.2$ pip install cryptography{code}
*Proposed fix:*

_Include the `cryptography` package in the setup / package requirements_

 

Having fixed that, the following issue occurs when trying to:
{code:java}
(venv) bash-3.2$ airflow initdb{code}
Exempt:
{code:java}
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 107, in get_fernet
raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect 
padding
[2018-08-08 10:50:50,697]

{models.py:643}

ERROR - Failed to load fernet while encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 105, in get_fernet
return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
File 
"~/project/airflow/venv/lib/python3.6/site-packages/cryptography/fernet.py", 
line 34, in _init_
key = base64.urlsafe_b64decode(key)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 133, in urlsafe_b64decode
return b64decode(s)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding{code}
Which after some googling leads to the conclusion that the 
~/project/airflow/airflow.cfg fernet_key field is not set to the correct value.

*Feature request:*

_Have the setup automatically generate a valid fernet key for the user._

The fact that this page exists: [https://bcb.github.io/airflow/fernet-key] 
suggests this could easily be a part of the package.

I understand that this project is in incubator phase, but I would say having a 
quick start that is not working as-is will discourage users from trying out 
this project. Thank you for considering.

  was:
Thank you for developing Airflow!

Having ran through the Quick Start, i've come across two issues that I would 
like to highlight:
{code:java}
bash-3.2$ cd ~/project/airflow/
bash-3.2$ export AIRFLOW_HOME=~/project/airflow
bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$ pip install --upgrade pip
(venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 17.7.0: 
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
(venv) bash-3.2$ python -V
Python 3.6.5
(venv) bash-3.2$ pip -V
pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
3.6)
(venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
Results in:
{code:java}
During handling of the above exception, another exception occurred:Traceback 
(most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File

[jira] [Created] (AIRFLOW-2873) Improvements to Quick Start flow

G. Geijteman created AIRFLOW-2873:
-

 Summary: Improvements to Quick Start flow
 Key: AIRFLOW-2873
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2873
 Project: Apache Airflow
  Issue Type: Improvement
  Components: configuration
Affects Versions: Airflow 1.9.0
Reporter: G. Geijteman


Thank you for developing Airflow!

Having ran through the Quick Start, i've come across two issues that I would 
like to highlight:
{code:java}
bash-3.2$ cd ~/project/airflow/
bash-3.2$ export AIRFLOW_HOME=~/project/airflow
bash-3.2$ python3 -m venv $AIRFLOW_HOME/venv
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$ pip install --upgrade pip
(venv) bash-3.2$ uname -a Darwin mac.local 17.7.0 Darwin Kernel Version 17.7.0: 
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
(venv) bash-3.2$ python -V
Python 3.6.5
(venv) bash-3.2$ pip -V
pip 18.0 from ~/project/airflow/venv/lib/python3.6/site-packages/pip (python 
3.6)
(venv) bash-3.2$ pip install apache-airflow[redis,postgres] -U {code}
Results in:
{code:java}
During handling of the above exception, another exception occurred:Traceback 
(most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 103, in get_fernet
raise AirflowException('Failed to import Fernet, it may not be installed')
airflow.exceptions.AirflowException: Failed to import Fernet, it may not be 
installed
[2018-08-08 10:49:01,121]{models.py:643}ERROR - Failed to load fernet while 
encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 101, in get_fernet
from cryptography.fernet import Fernet
ModuleNotFoundError: No module named 'cryptography'{code}
This is solved by:
{code:java}
(venv) bash-3.2$ pip install cryptography{code}
*Proposed fix:*

_Include the `cryptography` package_

 

Having fixed that, the following issue occurs when trying to:
{code:java}
(venv) bash-3.2$ airflow initdb{code}
Exempt:
{code:java}
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 639, in set_extra
fernet = get_fernet()
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 107, in get_fernet
raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect 
padding
[2018-08-08 10:50:50,697]

{models.py:643}

ERROR - Failed to load fernet while encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "~/project/airflow/venv/lib/python3.6/site-packages/airflow/models.py", 
line 105, in get_fernet
return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
File 
"~/project/airflow/venv/lib/python3.6/site-packages/cryptography/fernet.py", 
line 34, in _init_
key = base64.urlsafe_b64decode(key)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 133, in urlsafe_b64decode
return b64decode(s)
File 
"/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/base64.py",
 line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding{code}

 Which after some googling leads to the conclusion that the 
~/project/airflow/airflow.cfg fernet_key field is not set to the correct value.

*Feature request:*

_Have the setup automatically generate a valid fernet key for the user._

The fact that this page exists: [https://bcb.github.io/airflow/fernet-key] 
suggests this could easily be a part of the package.

I understand that this project is in incubator phase, but I would say having a 
quick start that is not working as-is will discourage users from trying out 
this project. Thank you for considering.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance

2018-08-08 Thread Bolke de Bruin (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572906#comment-16572906
 ] 

Bolke de Bruin edited comment on AIRFLOW-2870 at 8/8/18 9:05 AM:
-

or use with_entities, trying that. It's a very annoying migration. DagBags can 
be huge


was (Author: bolke):
or use with_entities, trying that

> Migrations fail when upgrading from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance
> 
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> 
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", 
> line 1182, in _execute_context
> context)
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", 
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance 
> from the current code version, which has changes to the task_instance table 
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an 
> executor_config column that does not exist as of when 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because 
> the migration branches on table existence/non-existence at a point that will 
> hide the issue from new installs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance

2018-08-08 Thread Bolke de Bruin (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572906#comment-16572906
 ] 

Bolke de Bruin commented on AIRFLOW-2870:
-

or use with_entities, trying that

> Migrations fail when upgrading from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance
> 
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> 
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", 
> line 1182, in _execute_context
> context)
>   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", 
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance 
> from the current code version, which has changes to the task_instance table 
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an 
> executor_config column that does not exist as of when 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because 
> the migration branches on table existence/non-existence at a point that will 
> hide the issue from new installs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment

2018-08-08 Thread Ash Berlin-Taylor (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572901#comment-16572901
 ] 

Ash Berlin-Taylor commented on AIRFLOW-2871:


Oh - I so we do. I guess what I'd like then is:

- latest to not be master, but the latest release.
- Nicer version numbers  (i.e. from the tags, not v1-9-stable etc)
- if a.i.a.org is to be kept the warning banner saying this is for master with 
a link to RTD.)

I don't know how possible any of that is.

> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe 
> this would require.  I am happy to take this issue if I can get that.  I have 
> experience as an admin of another project on RTD (simple-salesforce).
> 
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA

[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572898#comment-16572898
 ] 

Kaxil Naik commented on AIRFLOW-2871:
-

We have also listed this at our confluence page at 
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
 

>The site for the Airflow documentation that used to be located at 
>pythonhosted.org is now located at http://airflow.incubator.apache.org/ .This 
>should point to latest stable, while readthedocs.org keeps track of versioned 
>documentation.



> Harden and improve Read the Docs build environment
> --
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some 
> oddities in our Read the Docs (RTD) build environment especially around 
> cached dependencies.  This motivates hardening and showing some love to our 
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock 
> dependency discussed in PR #3703 above. I think that our RTD environment 
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --ignore-installed --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens 
> to have mock installed already.
> {code:java}
> python 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip 
> install --upgrade --cache-dir 
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip 
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - 
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - 
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - 
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - 
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first 
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in 
> our current config in incubator-airflow; I believe they are installed as 
> [core dependencies of RTD 
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further 
> down in the build.  In the case of mock, we were getting lucky that 
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version 
> which allowed the docs build to succeed.  (There might be other cases of 
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like 
> dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears 
> cache dir)
>  *** 
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** 
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe 
> this would require.  I am happy to take this issue if I can get that.  I have 
> experience as an admin of another project on RTD (simple-salesforce).
> 
> /cc Everyone who commented in PR #3703 - [~kaxilnaik],

[jira] [Updated] (AIRFLOW-2871) Harden and improve Read the Docs build environment