date:20181002

codecov-io edited a comment on issue #3992: [AIRFLOW-620] Feature to tail 
custom number of logs instead of rendering whole log
URL: 
https://github.com/apache/incubator-airflow/pull/3992#issuecomment-426519197
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=h1)
 Report
   > Merging 
[#3992](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `0.16%`.
   > The diff coverage is `41.11%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3992/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3992  +/-   ##
   ==
   - Coverage   75.49%   75.32%   -0.17% 
   ==
 Files 199  199  
 Lines   1594716023  +76 
   ==
   + Hits1203912070  +31 
   - Misses   3908 3953  +45
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `62.93% <0%> (-1.51%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=)
 | `77.14% <33.33%> (-12.27%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.76% <70%> (-0.09%)` | :arrow_down: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.89% <70%> (-0.15%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.48% <0%> (+0.35%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=footer).
 Last update 
[45ed3ce...054a9c8](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log

codecov-io commented on issue #3992: [AIRFLOW-620] Feature to tail custom 
number of logs instead of rendering whole log
URL: 
https://github.com/apache/incubator-airflow/pull/3992#issuecomment-426519197
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=h1)
 Report
   > Merging 
[#3992](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `0.16%`.
   > The diff coverage is `41.11%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3992/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3992  +/-   ##
   ==
   - Coverage   75.49%   75.32%   -0.17% 
   ==
 Files 199  199  
 Lines   1594716023  +76 
   ==
   + Hits1203912070  +31 
   - Misses   3908 3953  +45
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `62.93% <0%> (-1.51%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=)
 | `77.14% <33.33%> (-12.27%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.76% <70%> (-0.09%)` | :arrow_down: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.89% <70%> (-0.15%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.48% <0%> (+0.35%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=footer).
 Last update 
[45ed3ce...054a9c8](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page



[ 
https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636470#comment-16636470
 ] 

ASF GitHub Bot commented on AIRFLOW-620:


phani8996 opened a new pull request #3992: [AIRFLOW-620] Feature to tail custom 
number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992
 
 
   Dear Airflow Maintainers,
   
   This is a change in the way we render logs. This PR will give a dropdown of 
number of lines to be tailed with a refresh button. On refresh webserver fetch 
selected number of lines from local worker logs or remote worker logs. On can 
toggle this feature using config variable tail_logs in webserver, default 
number of lines to tail via config variable num_lines in webserver and items in 
dropdown can be controlled via config variable tail_lines_list in webserver in 
csv format
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-620
   
   Testing Done:
   
   Manually tested on all major browsers.
   This will add a dropdown of number of lines to be tailed and refresh button 
to get latest logs instead of refreshing the whole page.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Ability to refresh logs of a TI without reloading whole page
> 
>
> Key: AIRFLOW-620
> URL: https://issues.apache.org/jira/browse/AIRFLOW-620
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Sumit Maheshwari
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: UI, usability
>
> As of now to see the latest logs of a task instance one has to refresh the 
> whole page, which is time consuming. It'll be great if just logs can be 
> re-fetched from backend. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phani8996 opened a new pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log

phani8996 opened a new pull request #3992: [AIRFLOW-620] Feature to tail custom 
number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992
 
 
   Dear Airflow Maintainers,
   
   This is a change in the way we render logs. This PR will give a dropdown of 
number of lines to be tailed with a refresh button. On refresh webserver fetch 
selected number of lines from local worker logs or remote worker logs. On can 
toggle this feature using config variable tail_logs in webserver, default 
number of lines to tail via config variable num_lines in webserver and items in 
dropdown can be controlled via config variable tail_lines_list in webserver in 
csv format
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-620
   
   Testing Done:
   
   Manually tested on all major browsers.
   This will add a dropdown of number of lines to be tailed and refresh button 
to get latest logs instead of refreshing the whole page.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page



[ 
https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636465#comment-16636465
 ] 

ASF GitHub Bot commented on AIRFLOW-620:


phani8996 closed pull request #3988: [AIRFLOW-620] Add a dropdown and refresh 
button to tail selected number of lines from worker log
URL: https://github.com/apache/incubator-airflow/pull/3988
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 09bd0c1806..b5f7528332 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1023,17 +1023,41 @@ def scheduler(args):
 @cli_utils.action_logging
 def serve_logs(args):
 print("Starting flask")
-import flask
-flask_app = flask.Flask(__name__)
+from flask import Flask, request, Response, stream_with_context, 
send_from_directory
+flask_app = Flask(__name__)
 
 @flask_app.route('/log/')
 def serve_logs(filename):  # noqa
+def tail_logs(logdir, filename, num_lines):
+logpath = "{logdir}/{filename}".format(logdir=logdir, 
filename=filename)
+logsize = os.path.getsize(logpath)
+if logsize >= 100 * 1024 * 1024:
+p1 = subprocess.Popen(["tail", "-n " + str(num_lines), 
filename],
+  stdout=subprocess.PIPE, cwd=log)
+out, err = p1.communicate()
+out = "Tailing file\n\n" + out.decode("utf-8")
+else:
+fl = open("{log}//{filename}".format(log=log, 
filename=filename), "r")
+lines = fl.readlines()
+fl.close()
+out = "".join(l for l in lines[-num_lines:])
+line = "* Showing only last {num_lines} lines from {filename} 
*" \
+   "\n\n\n{out}".format(num_lines=num_lines, 
filename=filename, out=out)
+yield line
+num_lines = request.args.get("num_lines")
+try:
+num_lines = int(num_lines)
+except ValueError or TypeError:
+num_lines = None
 log = os.path.expanduser(conf.get('core', 'BASE_LOG_FOLDER'))
-return flask.send_from_directory(
-log,
-filename,
-mimetype="application/json",
-as_attachment=False)
+if num_lines:
+return Response(stream_with_context(tail_logs(log, filename, 
num_lines)))
+else:
+return send_from_directory(
+log,
+filename,
+mimetype="application/json",
+as_attachment=False)
 
 WORKER_LOG_SERVER_PORT = \
 int(conf.get('celery', 'WORKER_LOG_SERVER_PORT'))
diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index bb4ab208d7..4a603b1b54 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -317,6 +317,10 @@ default_dag_run_display_number = 25
 # Enable werkzeug `ProxyFix` middleware
 enable_proxy_fix = False
 
+# Flag to enable tailing logs
+tail_logs = True
+num_lines = 100
+tail_lines_list = 50,100,200,500,1000
 
 [email]
 email_backend = airflow.utils.email.send_email_smtp
diff --git a/airflow/utils/log/file_task_handler.py 
b/airflow/utils/log/file_task_handler.py
index 113bd254ad..803c00e02b 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -20,6 +20,7 @@
 import logging
 import os
 import requests
+import subprocess
 
 from airflow import configuration as conf
 from airflow.configuration import AirflowConfigException
@@ -94,17 +95,38 @@ def _read(self, ti, try_number, metadata=None):
 # is needed to get correct log path.
 log_relative_path = self._render_filename(ti, try_number)
 location = os.path.join(self.local_base, log_relative_path)
+if metadata and "num_lines" in metadata:
+num_lines = metadata.num_lines
+else:
+num_lines = None
 
 log = ""
 
 if os.path.exists(location):
-try:
-with open(location) as f:
-log += "*** Reading local file: {}\n".format(location)
-log += "".join(f.readlines())
-except Exception as e:
-log = "*** Failed to load local log file: 
{}\n".format(location)
-log += "*** {}\n".format(str(e))
+if num_lines:
+try:
+logsize = os.path.getsize(location)
+if logsize >= 100 * 1024 * 1024:
+p1 = subprocess.Popen(["tail", "-n " + str(num_lines), 
location],
+

[GitHub] phani8996 closed pull request #3988: [AIRFLOW-620] Add a dropdown and refresh button to tail selected number of lines from worker log

phani8996 closed pull request #3988: [AIRFLOW-620] Add a dropdown and refresh 
button to tail selected number of lines from worker log
URL: https://github.com/apache/incubator-airflow/pull/3988
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 09bd0c1806..b5f7528332 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1023,17 +1023,41 @@ def scheduler(args):
 @cli_utils.action_logging
 def serve_logs(args):
 print("Starting flask")
-import flask
-flask_app = flask.Flask(__name__)
+from flask import Flask, request, Response, stream_with_context, 
send_from_directory
+flask_app = Flask(__name__)
 
 @flask_app.route('/log/')
 def serve_logs(filename):  # noqa
+def tail_logs(logdir, filename, num_lines):
+logpath = "{logdir}/{filename}".format(logdir=logdir, 
filename=filename)
+logsize = os.path.getsize(logpath)
+if logsize >= 100 * 1024 * 1024:
+p1 = subprocess.Popen(["tail", "-n " + str(num_lines), 
filename],
+  stdout=subprocess.PIPE, cwd=log)
+out, err = p1.communicate()
+out = "Tailing file\n\n" + out.decode("utf-8")
+else:
+fl = open("{log}//{filename}".format(log=log, 
filename=filename), "r")
+lines = fl.readlines()
+fl.close()
+out = "".join(l for l in lines[-num_lines:])
+line = "* Showing only last {num_lines} lines from {filename} 
*" \
+   "\n\n\n{out}".format(num_lines=num_lines, 
filename=filename, out=out)
+yield line
+num_lines = request.args.get("num_lines")
+try:
+num_lines = int(num_lines)
+except ValueError or TypeError:
+num_lines = None
 log = os.path.expanduser(conf.get('core', 'BASE_LOG_FOLDER'))
-return flask.send_from_directory(
-log,
-filename,
-mimetype="application/json",
-as_attachment=False)
+if num_lines:
+return Response(stream_with_context(tail_logs(log, filename, 
num_lines)))
+else:
+return send_from_directory(
+log,
+filename,
+mimetype="application/json",
+as_attachment=False)
 
 WORKER_LOG_SERVER_PORT = \
 int(conf.get('celery', 'WORKER_LOG_SERVER_PORT'))
diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index bb4ab208d7..4a603b1b54 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -317,6 +317,10 @@ default_dag_run_display_number = 25
 # Enable werkzeug `ProxyFix` middleware
 enable_proxy_fix = False
 
+# Flag to enable tailing logs
+tail_logs = True
+num_lines = 100
+tail_lines_list = 50,100,200,500,1000
 
 [email]
 email_backend = airflow.utils.email.send_email_smtp
diff --git a/airflow/utils/log/file_task_handler.py 
b/airflow/utils/log/file_task_handler.py
index 113bd254ad..803c00e02b 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -20,6 +20,7 @@
 import logging
 import os
 import requests
+import subprocess
 
 from airflow import configuration as conf
 from airflow.configuration import AirflowConfigException
@@ -94,17 +95,38 @@ def _read(self, ti, try_number, metadata=None):
 # is needed to get correct log path.
 log_relative_path = self._render_filename(ti, try_number)
 location = os.path.join(self.local_base, log_relative_path)
+if metadata and "num_lines" in metadata:
+num_lines = metadata.num_lines
+else:
+num_lines = None
 
 log = ""
 
 if os.path.exists(location):
-try:
-with open(location) as f:
-log += "*** Reading local file: {}\n".format(location)
-log += "".join(f.readlines())
-except Exception as e:
-log = "*** Failed to load local log file: 
{}\n".format(location)
-log += "*** {}\n".format(str(e))
+if num_lines:
+try:
+logsize = os.path.getsize(location)
+if logsize >= 100 * 1024 * 1024:
+p1 = subprocess.Popen(["tail", "-n " + str(num_lines), 
location],
+  stdout=subprocess.PIPE)
+out, err = p1.communicate()
+log = "Tailing file\n\n" + out.decode("utf-8")
+else:
+fl = open(location,

[GitHub] codecov-io commented on issue #3991: [WIP] Backfill missing sensor tests.

codecov-io commented on issue #3991: [WIP] Backfill missing sensor tests.
URL: 
https://github.com/apache/incubator-airflow/pull/3991#issuecomment-426509904
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=h1)
 Report
   > Merging 
[#3991](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **increase** coverage by `0.12%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3991/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3991  +/-   ##
   ==
   + Coverage   75.49%   75.61%   +0.12% 
   ==
 Files 199  199  
 Lines   1594715945   -2 
   ==
   + Hits1203912057  +18 
   + Misses   3908 3888  -20
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3991/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=)
 | `100% <100%> (+9.52%)` | :arrow_up: |
   | 
[airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3991/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=)
 | `84.84% <100%> (+54.54%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=footer).
 Last update 
[45ed3ce...ab495b3](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3988: [AIRFLOW-620] Add a dropdown and refresh button to tail selected number of lines from worker log

codecov-io edited a comment on issue #3988: [AIRFLOW-620] Add a dropdown and 
refresh button to tail selected number of lines from worker log
URL: 
https://github.com/apache/incubator-airflow/pull/3988#issuecomment-426488241
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=h1)
 Report
   > Merging 
[#3988](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `60.33%`.
   > The diff coverage is `1.11%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3988/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#3988   +/-   ##
   ===
   - Coverage   75.49%   15.15%   -60.34% 
   ===
 Files 199  199   
 Lines   1594716023   +76 
   ===
   - Hits12039 2429 -9610 
   - Misses   390813594 +9686
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `0% <0%> (-68.86%)` | :arrow_down: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `13.51% <0%> (-50.92%)` | :arrow_down: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `0% <0%> (-72.05%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=)
 | `20.95% <3.7%> (-68.46%)` | :arrow_down: |
   | 
[airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [154 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=footer).
 Last update 
[45ed3ce...8b7db70](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jmcarp opened a new pull request #3991: [WIP] Backfill missing sensor tests.

jmcarp opened a new pull request #3991: [WIP] Backfill missing sensor tests.
URL: https://github.com/apache/incubator-airflow/pull/3991
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3990: [AIRFLOW-2951] Update dag_run table end_date when state change

codecov-io commented on issue #3990: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: 
https://github.com/apache/incubator-airflow/pull/3990#issuecomment-426505443
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=h1)
 Report
   > Merging 
[#3990](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=desc)
 into 
[v1-10-test](https://codecov.io/gh/apache/incubator-airflow/commit/0ee35e79adb0c3e1f66cc136adbe95b8ce5cc85b?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3990/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   v1-10-test#3990  +/-   ##
   ==
   + Coverage   77.77%   77.78%   +<.01% 
   ==
 Files 203  203  
 Lines   1530615307   +1 
   ==
   + Hits1190411906   +2 
   + Misses   3402 3401   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3990/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `88.76% <100%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=footer).
 Last update 
[0ee35e7...318f8f2](https://codecov.io/gh/apache/incubator-airflow/pull/3990?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] noronham commented on issue #3985: [AIRFLOW-3138] Use current data type for migrations

noronham commented on issue #3985: [AIRFLOW-3138] Use current data type for 
migrations
URL: 
https://github.com/apache/incubator-airflow/pull/3985#issuecomment-426501635
 
 
   I did a little more testing today with mysql, and while the mysql version 
I'm using doesn't currently set 
   explicit_defaults_for_timestamp (so I can't upgrade to current master at the 
moment) - the migration for this runs fine both with and without my diff.
   
   On an unrelated note since I was testing a bit more using master instead of 
1.10.0 - there seems to be two current heads for migrations which makes it a 
little harder to run upgrades and downgrades (you have to explicitly upgrade to 
cc1e65623dc7)
   9635ae0956e7 -> 0a2a5b66e19d (head), add task_reschedule table
   9635ae0956e7 -> dd25f486b8ea (head)
   
   Let me know if there is anything else I should/could do before this can be 
merged!
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2951) dag_run end_date Null after a dag is finished



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636379#comment-16636379
 ] 

ASF GitHub Bot commented on AIRFLOW-2951:
-

YingboWang opened a new pull request #3990: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: https://github.com/apache/incubator-airflow/pull/3990
 
 
   The existing airflow only change dag_run table end_date value when
   a user teminate a dag in web UI. The end_date will not be updated
   if airflow detected a dag finished and updated its state.
   
   This commit add end_date update in DagRun's set_state function to
   make up tho problem mentioned above.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following 
[Airflow-2951](https://issues.apache.org/jira/browse/AIRFLOW-2951) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2951
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Same feature was merged in master before. This PR is to backport to 
v1-10-test
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   tests/models.py:DagRunTest.test_dagrun_set_state_end_date
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   @ashb @KevinYang21 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> dag_run end_date Null after a dag is finished
> -
>
> Key: AIRFLOW-2951
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2951
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: DagRun
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Major
> Fix For: 1.10.1
>
>
> dag_run table should have an end_date updated when a dag is finished. 
> Currently only user activated dag termination request coming from UI may 
> change the "end_date" in dag_run table. All scheduled dags that are 
> automatically running by airflow will leave a NULL value after they fall into 
> a "success" or "failed" state. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] YingboWang opened a new pull request #3990: [AIRFLOW-2951] Update dag_run table end_date when state change

YingboWang opened a new pull request #3990: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: https://github.com/apache/incubator-airflow/pull/3990
 
 
   The existing airflow only change dag_run table end_date value when
   a user teminate a dag in web UI. The end_date will not be updated
   if airflow detected a dag finished and updated its state.
   
   This commit add end_date update in DagRun's set_state function to
   make up tho problem mentioned above.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following 
[Airflow-2951](https://issues.apache.org/jira/browse/AIRFLOW-2951) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2951
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Same feature was merged in master before. This PR is to backport to 
v1-10-test
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   tests/models.py:DagRunTest.test_dagrun_set_state_end_date
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   @ashb @KevinYang21 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3988: [AIRFLOW-620] Add a dropdown and refresh button to tail selected number of lines from worker log

codecov-io commented on issue #3988: [AIRFLOW-620] Add a dropdown and refresh 
button to tail selected number of lines from worker log
URL: 
https://github.com/apache/incubator-airflow/pull/3988#issuecomment-426488241
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=h1)
 Report
   > Merging 
[#3988](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `60.33%`.
   > The diff coverage is `1.11%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3988/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#3988   +/-   ##
   ===
   - Coverage   75.49%   15.15%   -60.34% 
   ===
 Files 199  199   
 Lines   1594716023   +76 
   ===
   - Hits12039 2429 -9610 
   - Misses   390813594 +9686
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `0% <0%> (-68.86%)` | :arrow_down: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `13.51% <0%> (-50.92%)` | :arrow_down: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `0% <0%> (-72.05%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=)
 | `20.95% <3.7%> (-68.46%)` | :arrow_down: |
   | 
[airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [154 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3988/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=footer).
 Last update 
[45ed3ce...53bf234](https://codecov.io/gh/apache/incubator-airflow/pull/3988?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3144) Validate kerberos keytab on startup



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kris Wilson updated AIRFLOW-3144:
-
Description: 
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos service principal keytab. Airflow happily 
accepted this file (which contained plain old ascii text) as a keytab and then 
broke at runtime with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
file to confirm it's validity on startup by shelling out to either `klist` or 
`kutil` (or using some equivalent lib).

 

  was:
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos service principal keytab. Airflow happily 
accepted this file (which contained plain old ascii text) as a keytab and then 
broke at runtime with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
file to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 


> Validate kerberos keytab on startup
> ---
>
> Key: AIRFLOW-3144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3144
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: authentication
>Reporter: Kris Wilson
>Priority: Minor
>
> at Twitter, we recently ran into an issue where an Airflow user was passing 
> the wrong secrets file as their kerberos service principal keytab. Airflow 
> happily accepted this file (which contained plain old ascii text) as a keytab 
> and then broke at runtime with the following opaque log message:
>  
> {code:java}
> [2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization 
> error for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
>  
> this made the problem unclear. rather than blindly accept any old file as a 
> keytab, it would be awesome if Airflow could run a validation step against 
> the file to confirm it's validity on startup by shelling out to either 
> `klist` or `kutil` (or using some equivalent lib).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3144) Validate kerberos keytab on startup



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kris Wilson updated AIRFLOW-3144:
-
Description: 
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos service principal keytab. Airflow happily 
accepted this file (which contained plain old ascii text) as a keytab and then 
broke at runtime with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
file to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 

  was:
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos service principal keytab. Airflow happily 
accepted this file (which contained plain old ascii text) as a keytab and then 
broke at runtime with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
keytab to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 


> Validate kerberos keytab on startup
> ---
>
> Key: AIRFLOW-3144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3144
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: authentication
>Reporter: Kris Wilson
>Priority: Minor
>
> at Twitter, we recently ran into an issue where an Airflow user was passing 
> the wrong secrets file as their kerberos service principal keytab. Airflow 
> happily accepted this file (which contained plain old ascii text) as a keytab 
> and then broke at runtime with the following opaque log message:
>  
> {code:java}
> [2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization 
> error for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
>  
> this made the problem unclear. rather than blindly accept any old file as a 
> keytab, it would be awesome if Airflow could run a validation step against 
> the file to confirm it's validity on startup by shelling out to either 
> `klist` or `kutil`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3144) Validate kerberos keytab on startup



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kris Wilson updated AIRFLOW-3144:
-
Description: 
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos service principal keytab. Airflow happily 
accepted this file (which contained plain old ascii text) as a keytab and then 
broke at runtime with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
keytab to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 

  was:
at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos keytab. Airflow happily accepted this file 
(which contained plain old ascii text) as a keytab and then broke at runtime 
with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
keytab to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 


> Validate kerberos keytab on startup
> ---
>
> Key: AIRFLOW-3144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3144
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: authentication
>Reporter: Kris Wilson
>Priority: Minor
>
> at Twitter, we recently ran into an issue where an Airflow user was passing 
> the wrong secrets file as their kerberos service principal keytab. Airflow 
> happily accepted this file (which contained plain old ascii text) as a keytab 
> and then broke at runtime with the following opaque log message:
>  
> {code:java}
> [2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization 
> error for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
>  
> this made the problem unclear. rather than blindly accept any old file as a 
> keytab, it would be awesome if Airflow could run a validation step against 
> the keytab to confirm it's validity on startup by shelling out to either 
> `klist` or `kutil`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3144) Validate kerberos keytab on startup

Kris Wilson created AIRFLOW-3144:


 Summary: Validate kerberos keytab on startup
 Key: AIRFLOW-3144
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3144
 Project: Apache Airflow
  Issue Type: Improvement
  Components: authentication
Reporter: Kris Wilson


at Twitter, we recently ran into an issue where an Airflow user was passing the 
wrong secrets file as their kerberos keytab. Airflow happily accepted this file 
(which contained plain old ascii text) as a keytab and then broke at runtime 
with the following opaque log message:

 
{code:java}
[2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization error 
for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
 

this made the problem unclear. rather than blindly accept any old file as a 
keytab, it would be awesome if Airflow could run a validation step against the 
keytab to confirm it's validity on startup by shelling out to either `klist` or 
`kutil`.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop

KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop 
from scheduler loop
URL: 
https://github.com/apache/incubator-airflow/pull/3873#issuecomment-426486693
 
 
   @feng-tao Any chance you got some spare time to give this change a shot :D?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] chang-asana commented on issue #3934: [AIRFLOW-3099] Stop Missing Section Errors for optional sections

chang-asana commented on issue #3934: [AIRFLOW-3099] Stop Missing Section 
Errors for optional sections
URL: 
https://github.com/apache/incubator-airflow/pull/3934#issuecomment-426480813
 
 
   @kaxil I think these additional sections can be added to the optional list: 
`operator`, `ldap`, `kerberos`, `dask`, `github_enterprise`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1945) Pass --autoscale to celery workers



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636314#comment-16636314
 ] 

ASF GitHub Bot commented on AIRFLOW-1945:
-

phani8996 opened a new pull request #3989: [AIRFLOW-1945] Autoscale celery 
workers for airflow added
URL: https://github.com/apache/incubator-airflow/pull/3989
 
 
   Dear Airflow Maintainers,
   
   This will add a provision to autoscale celery workers unlike same numbers of 
workers irrespective of number of running tasks.
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-1945
   
   Testing Done:
   
   Manually tested by passing arguments in cli


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Pass --autoscale to celery workers
> --
>
> Key: AIRFLOW-1945
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: celery, cli
>Reporter: Michael O.
>Assignee: Sai Phanindhra
>Priority: Trivial
>  Labels: easyfix
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Celery supports autoscaling of the worker pool size (number of tasks that can 
> parallelize within one worker node).  I'd like to propose to support passing 
> the --autoscale parameter to {{airflow worker}}.
> Since this is a trivial change, I am not sure if there's any reason for not 
> being supported already.(?)
> For example
> {{airflow worker --concurrency=4}} will set a fixed pool size of 4.
> With minimal changes in 
> [https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855]
>  it could support
> {{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10
> Some references:
> * 
> http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
> * 
> https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phani8996 opened a new pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added

phani8996 opened a new pull request #3989: [AIRFLOW-1945] Autoscale celery 
workers for airflow added
URL: https://github.com/apache/incubator-airflow/pull/3989
 
 
   Dear Airflow Maintainers,
   
   This will add a provision to autoscale celery workers unlike same numbers of 
workers irrespective of number of running tasks.
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-1945
   
   Testing Done:
   
   Manually tested by passing arguments in cli


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (AIRFLOW-2064) Polish timezone implementation

2018-10-02 Thread Sai Phanindhra (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636261#comment-16636261
 ] 

Sai Phanindhra edited comment on AIRFLOW-2064 at 10/3/18 12:13 AM:
---

Hi,

In point 2, if no timezone is specified we should ideally use UTC irrespective 
of timezone used by airflow scheduler. What should be changed here in that case?


was (Author: phani8996):
Hi,

In point 2, if no timezone is created we should ideally use UTC irrespective of 
timezone used by airflow scheduler. What should be changed here in that case?

> Polish timezone implementation
> --
>
> Key: AIRFLOW-2064
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2064
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Couple of things are left over after moving to time zone support:
>  
>  # End_dates within dags should be converted to UTC by using the time zone of 
> start_date if naive
>  # Task instances that are instantiated without timezone information for 
> their execution_date should convert those to UTC by using the DAG's timezone 
> or configured
>  # Some doc polishing
>  # Tests should be added that cover more of the edge cases



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2064) Polish timezone implementation

2018-10-02 Thread Sai Phanindhra (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636261#comment-16636261
 ] 

Sai Phanindhra commented on AIRFLOW-2064:
-

Hi,

In point 2, if no timezone is created we should ideally use UTC irrespective of 
timezone used by airflow scheduler. What should be changed here in that case?

> Polish timezone implementation
> --
>
> Key: AIRFLOW-2064
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2064
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Couple of things are left over after moving to time zone support:
>  
>  # End_dates within dags should be converted to UTC by using the time zone of 
> start_date if naive
>  # Task instances that are instantiated without timezone information for 
> their execution_date should convert those to UTC by using the DAG's timezone 
> or configured
>  # Some doc polishing
>  # Tests should be added that cover more of the edge cases



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page

2018-10-02 Thread Sai Phanindhra (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Phanindhra reassigned AIRFLOW-620:
--

Assignee: Sai Phanindhra  (was: Sumit Maheshwari)

> Ability to refresh logs of a TI without reloading whole page
> 
>
> Key: AIRFLOW-620
> URL: https://issues.apache.org/jira/browse/AIRFLOW-620
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Sumit Maheshwari
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: UI, usability
>
> As of now to see the latest logs of a task instance one has to refresh the 
> whole page, which is time consuming. It'll be great if just logs can be 
> re-fetched from backend. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page



[ 
https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636253#comment-16636253
 ] 

ASF GitHub Bot commented on AIRFLOW-620:


phani8996 opened a new pull request #3988: [AIRFLOW-620] Add a dropdown and 
refresh button to tail selected number of lines from worker log
URL: https://github.com/apache/incubator-airflow/pull/3988
 
 
   Dear Airflow Maintainers,
   
   This is a change in the way we render logs. This PR will give a dropdown of 
number of lines to be tailed with a refresh button. On refresh webserver fetch 
selected number of lines from local worker logs or remote worker logs. On can 
toggle this feature using config variable `tail_logs` in `webserver`, default 
number of lines to tail via config variable `num_lines ` in `webserver` and 
items in dropdown can be controlled via config variable `tail_lines_list ` in 
`webserver` in csv format
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-620
   
   **Testing Done:**
   
   Manually tested on all major browsers.
   `This will add a dropdown of number of lines to be tailed and refresh button 
to get latest logs instead of refreshing the whole page.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Ability to refresh logs of a TI without reloading whole page
> 
>
> Key: AIRFLOW-620
> URL: https://issues.apache.org/jira/browse/AIRFLOW-620
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>Priority: Major
>  Labels: UI, usability
>
> As of now to see the latest logs of a task instance one has to refresh the 
> whole page, which is time consuming. It'll be great if just logs can be 
> re-fetched from backend. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phani8996 opened a new pull request #3988: [AIRFLOW-620] Add a dropdown and refresh button to tail selected number of lines from worker log

phani8996 opened a new pull request #3988: [AIRFLOW-620] Add a dropdown and 
refresh button to tail selected number of lines from worker log
URL: https://github.com/apache/incubator-airflow/pull/3988
 
 
   Dear Airflow Maintainers,
   
   This is a change in the way we render logs. This PR will give a dropdown of 
number of lines to be tailed with a refresh button. On refresh webserver fetch 
selected number of lines from local worker logs or remote worker logs. On can 
toggle this feature using config variable `tail_logs` in `webserver`, default 
number of lines to tail via config variable `num_lines ` in `webserver` and 
items in dropdown can be controlled via config variable `tail_lines_list ` in 
`webserver` in csv format
   
   Please accept this PR that addresses the following issues:
   https://issues.apache.org/jira/browse/AIRFLOW-620
   
   **Testing Done:**
   
   Manually tested on all major browsers.
   `This will add a dropdown of number of lines to be tailed and refresh button 
to get latest logs instead of refreshing the whole page.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3134) Add New Operator - MySQLToS3TransformOperator

2018-10-02 Thread Anonymous (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anonymous reassigned AIRFLOW-3134:
--

Assignee: (was: Stefano Francavilla)

> Add New Operator - MySQLToS3TransformOperator
> -
>
> Key: AIRFLOW-3134
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3134
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 1.10.0
>Reporter: Stefano Francavilla
>Priority: Minor
>  Labels: MissingFeature, operators
>
> Taking inspiration from the [S3Transform 
> Operator|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/s3_file_transform_operator.py]
>  and from an use case I was working in the past weeks, I was wondering if it 
> would be useful to add a new operator: "MySQLToS3TransformOperator".
> The operator allows to transfer (transformed) data resulting from a SELECT 
> statement to an S3 Bucket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (AIRFLOW-3142) Fix deprecated calls to BigQueryBaseCursor.run_query

2018-10-02 Thread Jason Shao (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-3142 started by Jason Shao.
---
> Fix deprecated calls to BigQueryBaseCursor.run_query
> 
>
> Key: AIRFLOW-3142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3142
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Jason Shao
>Assignee: Jason Shao
>Priority: Minor
>
> BigQueryOperator.execute and BigQueryCursor.execute use `bql` instead of `sql`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator

codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes 
tolerations to kubernetes pod operator
URL: 
https://github.com/apache/incubator-airflow/pull/3806#issuecomment-416340975
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=h1)
 Report
   > Merging 
[#3806](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3806/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3806  +/-   ##
   ==
   - Coverage   75.49%   75.48%   -0.01% 
   ==
 Files 199  199  
 Lines   1594715947  
   ==
   - Hits1203912038   -1 
   - Misses   3908 3909   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.62% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=footer).
 Last update 
[45ed3ce...a193629](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3142) Fix deprecated calls to BigQueryBaseCursor.run_query



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636223#comment-16636223
 ] 

ASF GitHub Bot commented on AIRFLOW-3142:
-

jason-udacity opened a new pull request #3987: [AIRFLOW-3142] Fix deprecated 
calls to BigQueryBaseCursor.run_query
URL: https://github.com/apache/incubator-airflow/pull/3987
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix deprecated calls to BigQueryBaseCursor.run_query
> 
>
> Key: AIRFLOW-3142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3142
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Jason Shao
>Assignee: Jason Shao
>Priority: Minor
>
> BigQueryOperator.execute and BigQueryCursor.execute use `bql` instead of `sql`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] jason-udacity opened a new pull request #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query

jason-udacity opened a new pull request #3987: [AIRFLOW-3142] Fix deprecated 
calls to BigQueryBaseCursor.run_query
URL: https://github.com/apache/incubator-airflow/pull/3987
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator

codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes 
tolerations to kubernetes pod operator
URL: 
https://github.com/apache/incubator-airflow/pull/3806#issuecomment-416340975
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=h1)
 Report
   > Merging 
[#3806](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc)
 will **decrease** coverage by `60.26%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3806/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#3806   +/-   ##
   ===
   - Coverage   75.49%   15.22%   -60.27% 
   ===
 Files 199  199   
 Lines   1594715947   
   ===
   - Hits12039 2428 -9611 
   - Misses   390813519 +9611
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_task.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFzay5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/sensors/time\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3RpbWVfc2Vuc29yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [150 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=footer).
 Last update 
[45ed3ce...a193629](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3143) Support auto-zone in DataprocClusterCreateOperator

2018-10-02 Thread Wilson Lian (JIRA)

Wilson Lian created AIRFLOW-3143:


 Summary: Support auto-zone in DataprocClusterCreateOperator
 Key: AIRFLOW-3143
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3143
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib, operators
Reporter: Wilson Lian


[Dataproc 
Auto-zone|https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone]
 allows users to omit the zone when creating a cluster, and the service will 
pick a zone in the chosen region.

Providing an empty string or None for `zone` would match up with how users 
would request auto-zone via direct API access, but as-is the 
DataprocClusterCreateOperator makes a bad API request when such values are 
passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3142) Fix deprecated calls to BigQueryBaseCursor.run_query

2018-10-02 Thread Jason Shao (JIRA)

Jason Shao created AIRFLOW-3142:
---

 Summary: Fix deprecated calls to BigQueryBaseCursor.run_query
 Key: AIRFLOW-3142
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3142
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Jason Shao
Assignee: Jason Shao


BigQueryOperator.execute and BigQueryCursor.execute use `bql` instead of `sql`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] justinholmes edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator

justinholmes edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes 
tolerations to kubernetes pod operator
URL: 
https://github.com/apache/incubator-airflow/pull/3806#issuecomment-426458078
 
 
   @Fokko Are you happy to merge?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] justinholmes commented on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator

justinholmes commented on issue #3806: [AIRFLOW-2956] added kubernetes 
tolerations to kubernetes pod operator
URL: 
https://github.com/apache/incubator-airflow/pull/3806#issuecomment-426458078
 
 
   @Fokko Are you guys happy to merge?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] justinholmes commented on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator

justinholmes commented on issue #3806: [AIRFLOW-2956] added kubernetes 
tolerations to kubernetes pod operator
URL: 
https://github.com/apache/incubator-airflow/pull/3806#issuecomment-426457356
 
 
   Merged and Squashed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3798: [AIRFLOW-2951] Update dag_run table end_date when state change

ashb commented on issue #3798: [AIRFLOW-2951] Update dag_run table end_date 
when state change
URL: 
https://github.com/apache/incubator-airflow/pull/3798#issuecomment-426417370
 
 
   Please create a new pr (targeting the same Jira) against v1-10-test instead 
of matte master.
   
   That is, in your checkout create a new branch off the v1-10-test and then 
cherry pick or manually recreate the appropriate changes, commit (using 
--commit-from to get same message and time stamp if done manually) the push up 
for PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] neil90 commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file

neil90 commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow 
compression of file
URL: 
https://github.com/apache/incubator-airflow/pull/3893#issuecomment-426363816
 
 
   Hi @xnuinside , sorry about that I have been swamped at work I will finish 
this task by the weekend my apologies.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] r39132 commented on issue #3918: [AIRFLOW-3088] Include slack-compatible emoji image.

r39132 commented on issue #3918: [AIRFLOW-3088] Include slack-compatible emoji 
image.
URL: 
https://github.com/apache/incubator-airflow/pull/3918#issuecomment-426349153
 
 
   @sethwoodworth any updates?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] YingboWang commented on issue #3798: [AIRFLOW-2951] Update dag_run table end_date when state change

YingboWang commented on issue #3798: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: 
https://github.com/apache/incubator-airflow/pull/3798#issuecomment-426347360
 
 
   @ashb @KevinYang21 I would be happy to add this feature to v1.10 however I'm 
not familiar with the backport process. Should I cherry pick from base repo and 
create a PR to the test branch? Or I should start from my fork repo. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3137) Make ProxyFix middleware optional



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635676#comment-16635676
 ] 

ASF GitHub Bot commented on AIRFLOW-3137:
-

ashb closed pull request #3983: [AIRFLOW-3137] Make ProxyFix middleware 
optional.
URL: https://github.com/apache/incubator-airflow/pull/3983
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index 0028d7832f..bb4ab208d7 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -314,6 +314,9 @@ navbar_color = #007A87
 # Default dagrun to show in UI
 default_dag_run_display_number = 25
 
+# Enable werkzeug `ProxyFix` middleware
+enable_proxy_fix = False
+
 
 [email]
 email_backend = airflow.utils.email.send_email_smtp
diff --git a/airflow/www/app.py b/airflow/www/app.py
index a7f7424cb6..98e3003908 100644
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -47,7 +47,8 @@ def create_app(config=None, testing=False):
 log = LoggingMixin().log
 
 app = Flask(__name__)
-app.wsgi_app = ProxyFix(app.wsgi_app)
+if configuration.conf.getboolean('webserver', 'ENABLE_PROXY_FIX'):
+app.wsgi_app = ProxyFix(app.wsgi_app)
 app.secret_key = configuration.conf.get('webserver', 'SECRET_KEY')
 app.config['LOGIN_DISABLED'] = not configuration.conf.getboolean(
 'webserver', 'AUTHENTICATE')
diff --git a/airflow/www_rbac/app.py b/airflow/www_rbac/app.py
index 321185ee9b..392dce1b31 100644
--- a/airflow/www_rbac/app.py
+++ b/airflow/www_rbac/app.py
@@ -41,7 +41,8 @@
 def create_app(config=None, session=None, testing=False, app_name="Airflow"):
 global app, appbuilder
 app = Flask(__name__)
-app.wsgi_app = ProxyFix(app.wsgi_app)
+if conf.getboolean('webserver', 'ENABLE_PROXY_FIX'):
+app.wsgi_app = ProxyFix(app.wsgi_app)
 app.secret_key = conf.get('webserver', 'SECRET_KEY')
 
 airflow_home_path = conf.get('core', 'AIRFLOW_HOME')
diff --git a/docs/integration.rst b/docs/integration.rst
index f1ad6c5cb6..6ef7bd8398 100644
--- a/docs/integration.rst
+++ b/docs/integration.rst
@@ -65,6 +65,15 @@ Your reverse proxy (ex: nginx) should be configured as 
follow:
   }
   }
 
+To ensure that Airflow generates URLs with the correct scheme when
+running behind a TLS-terminating proxy, you should configure the proxy
+to set the `X-Forwarded-Proto` header, and enable the `ProxyFix`
+middleware in your `airflow.cfg`::
+
+enable_proxy_fix = True
+
+Note: you should only enable the `ProxyFix` middleware when running
+Airflow behind a trusted proxy (AWS ELB, nginx, etc.).
 
 .. _Azure:
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make ProxyFix middleware optional
> -
>
> Key: AIRFLOW-3137
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3137
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Josh Carp
>Assignee: Josh Carp
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The werkzeug ProxyFix middleware should only be used when the app is run 
> behind a trusted proxy. We should enable ProxyFix via a configuration flag, 
> like in superset.
> From the werkzeug docs: "Do not use this middleware in non-proxy setups for 
> security reasons."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb closed pull request #3983: [AIRFLOW-3137] Make ProxyFix middleware optional.

ashb closed pull request #3983: [AIRFLOW-3137] Make ProxyFix middleware 
optional.
URL: https://github.com/apache/incubator-airflow/pull/3983
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index 0028d7832f..bb4ab208d7 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -314,6 +314,9 @@ navbar_color = #007A87
 # Default dagrun to show in UI
 default_dag_run_display_number = 25
 
+# Enable werkzeug `ProxyFix` middleware
+enable_proxy_fix = False
+
 
 [email]
 email_backend = airflow.utils.email.send_email_smtp
diff --git a/airflow/www/app.py b/airflow/www/app.py
index a7f7424cb6..98e3003908 100644
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -47,7 +47,8 @@ def create_app(config=None, testing=False):
 log = LoggingMixin().log
 
 app = Flask(__name__)
-app.wsgi_app = ProxyFix(app.wsgi_app)
+if configuration.conf.getboolean('webserver', 'ENABLE_PROXY_FIX'):
+app.wsgi_app = ProxyFix(app.wsgi_app)
 app.secret_key = configuration.conf.get('webserver', 'SECRET_KEY')
 app.config['LOGIN_DISABLED'] = not configuration.conf.getboolean(
 'webserver', 'AUTHENTICATE')
diff --git a/airflow/www_rbac/app.py b/airflow/www_rbac/app.py
index 321185ee9b..392dce1b31 100644
--- a/airflow/www_rbac/app.py
+++ b/airflow/www_rbac/app.py
@@ -41,7 +41,8 @@
 def create_app(config=None, session=None, testing=False, app_name="Airflow"):
 global app, appbuilder
 app = Flask(__name__)
-app.wsgi_app = ProxyFix(app.wsgi_app)
+if conf.getboolean('webserver', 'ENABLE_PROXY_FIX'):
+app.wsgi_app = ProxyFix(app.wsgi_app)
 app.secret_key = conf.get('webserver', 'SECRET_KEY')
 
 airflow_home_path = conf.get('core', 'AIRFLOW_HOME')
diff --git a/docs/integration.rst b/docs/integration.rst
index f1ad6c5cb6..6ef7bd8398 100644
--- a/docs/integration.rst
+++ b/docs/integration.rst
@@ -65,6 +65,15 @@ Your reverse proxy (ex: nginx) should be configured as 
follow:
   }
   }
 
+To ensure that Airflow generates URLs with the correct scheme when
+running behind a TLS-terminating proxy, you should configure the proxy
+to set the `X-Forwarded-Proto` header, and enable the `ProxyFix`
+middleware in your `airflow.cfg`::
+
+enable_proxy_fix = True
+
+Note: you should only enable the `ProxyFix` middleware when running
+Airflow behind a trusted proxy (AWS ELB, nginx, etc.).
 
 .. _Azure:
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3137) Make ProxyFix middleware optional



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3137.

   Resolution: Fixed
Fix Version/s: 2.0.0

> Make ProxyFix middleware optional
> -
>
> Key: AIRFLOW-3137
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3137
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Josh Carp
>Assignee: Josh Carp
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The werkzeug ProxyFix middleware should only be used when the app is run 
> behind a trusted proxy. We should enable ProxyFix via a configuration flag, 
> like in superset.
> From the werkzeug docs: "Do not use this middleware in non-proxy setups for 
> security reasons."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb commented on issue #3985: [AIRFLOW-3138] Use current data type for migrations

ashb commented on issue #3985: [AIRFLOW-3138] Use current data type for 
migrations
URL: 
https://github.com/apache/incubator-airflow/pull/3985#issuecomment-426313412
 
 
   Oh, I was getting the migrations confused. I thought this was the one where 
we changed the timezone columns from native to UtcDateTime. But this  is an 
earlier one right?
   
   In which case yes this makes sense, and thank you for the detailed 
explanation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] noronham commented on issue #3985: [AIRFLOW-3138] Use current data type for migrations

noronham commented on issue #3985: [AIRFLOW-3138] Use current data type for 
migrations
URL: 
https://github.com/apache/incubator-airflow/pull/3985#issuecomment-426311793
 
 
   @ashb - the problem I ran into was that (at least for postgres) - the data 
was being read from a naive timestamp column in the db (pre 
0e2a74e0fc9f_add_time_zone_awareness.py and ) and so the python code was 
converting it to an assumed UTC date. The session.merge of the task instance 
then fails because (at least in postgres) the timestamp with timezone is passed 
into the query as a full string and so the key no longer matches the database 
and you get an error and the migration fails:
   **'sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 
'task_instance' expected to update 1 row(s); 0 were matched.'**
   
   I don't think that the migration really needs the date - it just needs the 
primary key and so while for runtime code/etc - I agree 100% that the 
UtcDateTime should be used. However, I believe in this case since the database 
during this migration is storing naive dates AND we are only using the date as 
a value type - it should be safe  to keep naive in the migration code.
   
   I'll see if I can test with mysql and confirm this doesn't cause any new 
errors.
 * install airflow 1.8 or 1.9
 * run a task (so task_instance table isn't empty)
 * upgrade to 1.10
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3984: [AIRFLOW-3141] Handle duration for missing dag.

codecov-io commented on issue #3984: [AIRFLOW-3141] Handle duration for missing 
dag.
URL: 
https://github.com/apache/incubator-airflow/pull/3984#issuecomment-426309110
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=h1)
 Report
   > Merging 
[#3984](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3984/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #3984  +/-   ##
   =
   - Coverage75.5%   75.5%   -0.01% 
   =
 Files 199 199  
 Lines   15945   15948   +3 
   =
   + Hits12039   12041   +2 
   - Misses   39063907   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `68.91% <100%> (+0.05%)` | :arrow_up: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.62% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=footer).
 Last update 
[6aabdf1...3566510](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3983: [AIRFLOW-3137] Make ProxyFix middleware optional.

codecov-io edited a comment on issue #3983: [AIRFLOW-3137] Make ProxyFix 
middleware optional.
URL: 
https://github.com/apache/incubator-airflow/pull/3983#issuecomment-426019072
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=h1)
 Report
   > Merging 
[#3983](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `50%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3983/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3983  +/-   ##
   ==
   + Coverage75.5%   75.51%   +0.01% 
   ==
 Files 199  199  
 Lines   1594515947   +2 
   ==
   + Hits1203912043   +4 
   + Misses   3906 3904   -2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3983/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `98.95% <50%> (-1.05%)` | :arrow_down: |
   | 
[airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3983/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=)
 | `96.7% <50%> (-1.08%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3983/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.48% <0%> (+0.35%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=footer).
 Last update 
[6aabdf1...96c8e5b](https://codecov.io/gh/apache/incubator-airflow/pull/3983?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jmcarp commented on a change in pull request #3983: [AIRFLOW-3137] Make ProxyFix middleware optional.

jmcarp commented on a change in pull request #3983: [AIRFLOW-3137] Make 
ProxyFix middleware optional.
URL: https://github.com/apache/incubator-airflow/pull/3983#discussion_r221970283
 
 

 ##
 File path: docs/integration.rst
 ##
 @@ -65,6 +65,12 @@ Your reverse proxy (ex: nginx) should be configured as 
follow:
   }
   }
 
+To ensure that Airflow generates URLs with the correct scheme when
+running behind a TLS-terminating proxy, you should configure the proxy
+to set the `X-Forwarded-Proto` header, and enable the `ProxyFix`
+middleware in your `airflow.cfg`::
+
+enable_proxy_fix = True
 
 Review comment:
   Updated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3141) Fix 500 on duration view when dag doesn't exist

2018-10-02 Thread Josh Carp (JIRA)

Josh Carp created AIRFLOW-3141:
--

 Summary: Fix 500 on duration view when dag doesn't exist
 Key: AIRFLOW-3141
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3141
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Josh Carp
Assignee: Josh Carp


Loading the duration view for a dag that doesn't exist throws a 500. Based on 
the behavior of other dag views, this should redirect to the admin view and 
flash an error message instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb commented on a change in pull request #3983: [AIRFLOW-3137] Make ProxyFix middleware optional.

ashb commented on a change in pull request #3983: [AIRFLOW-3137] Make ProxyFix 
middleware optional.
URL: https://github.com/apache/incubator-airflow/pull/3983#discussion_r221967217
 
 

 ##
 File path: docs/integration.rst
 ##
 @@ -65,6 +65,12 @@ Your reverse proxy (ex: nginx) should be configured as 
follow:
   }
   }
 
+To ensure that Airflow generates URLs with the correct scheme when
+running behind a TLS-terminating proxy, you should configure the proxy
+to set the `X-Forwarded-Proto` header, and enable the `ProxyFix`
+middleware in your `airflow.cfg`::
+
+enable_proxy_fix = True
 
 Review comment:
   Can we add a note here saying only enable this when you do have a trusted 
proxy in front of Airflow (i.e. nginx, AWS ELB etc.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jmcarp commented on issue #3983: [AIRFLOW-3137] Make ProxyFix middleware optional.

jmcarp commented on issue #3983: [AIRFLOW-3137] Make ProxyFix middleware 
optional.
URL: 
https://github.com/apache/incubator-airflow/pull/3983#issuecomment-426288426
 
 
   Good catch @ashb; I just updated the PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator

codecov-io edited a comment on issue #3961: [AIRFLOW-2794] Add 
WasbDeleteBlobOperator
URL: 
https://github.com/apache/incubator-airflow/pull/3961#issuecomment-425815697
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=h1)
 Report
   > Merging 
[#3961](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3961/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3961  +/-   ##
   ==
   + Coverage75.5%   75.54%   +0.03% 
   ==
 Files 199  199  
 Lines   1594515945  
   ==
   + Hits1203912045   +6 
   + Misses   3906 3900   -6
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3961/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.62% <0%> (-0.05%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3961/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.74% <0%> (+0.61%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=footer).
 Last update 
[6aabdf1...2901e32](https://codecov.io/gh/apache/incubator-airflow/pull/3961?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators

wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 
operators
URL: 
https://github.com/apache/incubator-airflow/pull/3828#issuecomment-426265075
 
 
   Yes, I do. I was away and will resume work to close this out. Thanks.
   
   On Tue, Oct 2, 2018, 7:48 AM Iuliia Volkova 
   wrote:
   
   > @wmorris75 , do you plan update PR and fix
   > comment's issues?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator

bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add 
WasbDeleteBlobOperator
URL: https://github.com/apache/incubator-airflow/pull/3961#discussion_r221909307
 
 

 ##
 File path: airflow/contrib/operators/wasb_delete_blob_operator.py
 ##
 @@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+from airflow.contrib.hooks.wasb_hook import WasbHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class WasbDeleteBlobOperator(BaseOperator):
+"""
+Deletes blob(s) on Azure Blob Storage.
+
+:param container_name: Name of the container. (templated)
+:type container_name: str
+:param blob_name: Name of the blob. (templated)
+:type blob_name: str
+:param wasb_conn_id: Reference to the wasb connection. (templated)
+:type wasb_conn_id: str
+:param check_options: Optional keyword arguments that
+`WasbHook.check_for_blob()` takes.
+:param is_prefix: If blob_name is a prefix, delete all files matching 
prefix.
+:type is_prefix: bool
+:param ignore_if_missing: if True, then return success even if the
+blob does not exist.
+:type ignore_if_missing: bool
+"""
+
+template_fields = ('container_name', 'blob_name', 'wasb_conn_id')
 
 Review comment:
   @ashb I have removed 'wasb_conn_id' from the list of template_fields, to 
make it more in line with the rest of the WasbOperators.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3140) Add MongoDBToGoogleStorage Opearator

2018-10-02 Thread jack (JIRA)

jack created AIRFLOW-3140:
-

 Summary: Add MongoDBToGoogleStorage Opearator
 Key: AIRFLOW-3140
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3140
 Project: Apache Airflow
  Issue Type: Task
Affects Versions: 1.10.0
Reporter: jack


Airflow has Mongo Hook

[https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/mongo_hook.py]

 

Please also add operator that transfer from MongoDB to Google Storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2618) Improve UI by add "Next Run" column

2018-10-02 Thread jack (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635352#comment-16635352
 ] 

jack commented on AIRFLOW-2618:
---

This ticket is a UI view of this CLI command: 
https://github.com/apache/incubator-airflow/pull/3834

> Improve UI by add "Next Run" column
> ---
>
> Key: AIRFLOW-2618
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2618
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: jack
>Priority: Minor
>
> Please add also a column in the UI for "Next Run". Ideally when passing mouse 
> over it we will also see the 5 next scheduled runs.
> This can be very helpful.
> If for some reason you think this is an "overhead" why not adding it and 
> allow a "personalized UI" feature where the user can set if this column will 
> appear or not. This can be a very good feature in allowing users 
> personalizing their own UI columns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators

codecov-io edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push 
flag for contrib's operators
URL: 
https://github.com/apache/incubator-airflow/pull/3981#issuecomment-425858829
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=h1)
 Report
   > Merging 
[#3981](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3981/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #3981   +/-   ##
   ==
 Coverage75.5%   75.5%   
   ==
 Files 199 199   
 Lines   15945   15945   
   ==
 Hits12039   12039   
 Misses   39063906
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=footer).
 Last update 
[6aabdf1...b552bb5](https://codecov.io/gh/apache/incubator-airflow/pull/3981?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3894: [AIRFLOW-3055] add get_dataset and get_datasets_list to bigquery_hook

xnuinside commented on issue #3894: [AIRFLOW-3055] add get_dataset and 
get_datasets_list to bigquery_hook
URL: 
https://github.com/apache/incubator-airflow/pull/3894#issuecomment-426245594
 
 
   @kaxil , I very quick.. :D please review


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3894: [AIRFLOW-3055] add get_dataset and get_datasets_list to bigquery_hook

codecov-io edited a comment on issue #3894: [AIRFLOW-3055] add get_dataset and 
get_datasets_list to bigquery_hook
URL: 
https://github.com/apache/incubator-airflow/pull/3894#issuecomment-423291432
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=h1)
 Report
   > Merging 
[#3894](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3894/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3894  +/-   ##
   ==
   + Coverage75.5%   75.52%   +0.02% 
   ==
 Files 199  199  
 Lines   1594515945  
   ==
   + Hits1203912043   +4 
   + Misses   3906 3902   -4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3894/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.48% <0%> (+0.35%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=footer).
 Last update 
[6aabdf1...9330a5b](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3894: [AIRFLOW-3055] add get_dataset and get_datasets_list to bigquery_hook

codecov-io edited a comment on issue #3894: [AIRFLOW-3055] add get_dataset and 
get_datasets_list to bigquery_hook
URL: 
https://github.com/apache/incubator-airflow/pull/3894#issuecomment-423291432
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=h1)
 Report
   > Merging 
[#3894](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3894/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3894  +/-   ##
   ==
   + Coverage75.5%   75.52%   +0.02% 
   ==
 Files 199  199  
 Lines   1594515945  
   ==
   + Hits1203912043   +4 
   + Misses   3906 3902   -4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3894/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.48% <0%> (+0.35%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=footer).
 Last update 
[6aabdf1...9330a5b](https://codecov.io/gh/apache/incubator-airflow/pull/3894?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators

xnuinside commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 
operators
URL: 
https://github.com/apache/incubator-airflow/pull/3828#issuecomment-426244028
 
 
   @wmorris75, do you plan update PR and fix comment's issues? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3885: [AIRFLOW-3001] Add index 'ti_dag_date' to taskinstance

xnuinside commented on issue #3885: [AIRFLOW-3001] Add index 'ti_dag_date' to 
taskinstance
URL: 
https://github.com/apache/incubator-airflow/pull/3885#issuecomment-426243450
 
 
   @ubermen , you need to squash all commits in one. @Fokko , any comments? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2933) Enable Codecov on Docker CI



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635337#comment-16635337
 ] 

Iuliia Volkova commented on AIRFLOW-2933:
-

[~Fokko], issue still open.. PR already merged

> Enable Codecov on Docker CI
> ---
>
> Key: AIRFLOW-2933
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2933
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>Priority: Major
>
> Right now the Codecov plugin is not working on the docker-ci that we're using 
> right now. This has to be fixed so we can track code coverage over time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file

xnuinside commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - 
allow compression of file
URL: 
https://github.com/apache/incubator-airflow/pull/3893#issuecomment-426242707
 
 
   @neil90, Are you planning to update the PR with adding tests? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator

bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add 
WasbDeleteBlobOperator
URL: https://github.com/apache/incubator-airflow/pull/3961#discussion_r221909307
 
 

 ##
 File path: airflow/contrib/operators/wasb_delete_blob_operator.py
 ##
 @@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+from airflow.contrib.hooks.wasb_hook import WasbHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class WasbDeleteBlobOperator(BaseOperator):
+"""
+Deletes blob(s) on Azure Blob Storage.
+
+:param container_name: Name of the container. (templated)
+:type container_name: str
+:param blob_name: Name of the blob. (templated)
+:type blob_name: str
+:param wasb_conn_id: Reference to the wasb connection. (templated)
+:type wasb_conn_id: str
+:param check_options: Optional keyword arguments that
+`WasbHook.check_for_blob()` takes.
+:param is_prefix: If blob_name is a prefix, delete all files matching 
prefix.
+:type is_prefix: bool
+:param ignore_if_missing: if True, then return success even if the
+blob does not exist.
+:type ignore_if_missing: bool
+"""
+
+template_fields = ('container_name', 'blob_name', 'wasb_conn_id')
 
 Review comment:
   I have removed 'wasb_conn_id' from the list of template_fields.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3948: [AIRFLOW-3071] clear Val of Variable from the UI

ashb commented on issue #3948: [AIRFLOW-3071] clear Val of Variable from the UI
URL: 
https://github.com/apache/incubator-airflow/pull/3948#issuecomment-426220390
 
 
   The code change looks sensible, but a couple of things:
   
   1. We should add a test that covers this - probably in 
tests/www/test_views.py as a new test function in `TestVariableView`
   2. Does this fix the form or is there a validation error saying value is 
required?
   3. (Annoyingly, right now) there are two WWW versions that need updating. 
There is also www_rbac (which will become the only version, but for now please 
update both versions and both tests)
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3123) Allow nested use of DAG as a context manager



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3123.

   Resolution: Fixed
Fix Version/s: 2.0.0

> Allow nested use of  DAG as a context manager
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
> Fix For: 2.0.0
>
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
>   op1 = Operator()
>   with dag:
> op2 = Operator()
>   op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the 
> nested context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (AIRFLOW-3123) Allow nested use of DAG as a context manager



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor reopened AIRFLOW-3123:


Re-opening so I can assign fixed version. (Please use "Resolved" rather than 
"Closed" in future - we can edit Resolved tickets but not closed ones.)

> Allow nested use of  DAG as a context manager
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
>   op1 = Operator()
>   with dag:
> op2 = Operator()
>   op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the 
> nested context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-3124) Broken webserver debug mode (RBAC)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3124.

   Resolution: Fixed
Fix Version/s: 1.10.1

> Broken webserver debug mode (RBAC)
> --
>
> Key: AIRFLOW-3124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3124
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp, webserver
>Reporter: Aaron Kosel
>Assignee: Aaron Kosel
>Priority: Minor
> Fix For: 1.10.1
>
>
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 7, in 
> exec(compile(f.read(), __file__, 'exec'))
> File "/airflow/airflow/bin/airflow", line 32, in 
> args.func(args)
> File "/airflow/airflow/utils/cli.py", line 74, in wrapper
> return f(*args, **kwargs)
> File "/airflow/airflow/bin/cli.py", line 875, in webserver
> app.run(debug=True, port=args.port, host=args.hostname,
> AttributeError: 'tuple' object has no attribute 'run'
> {code}
> Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204, 
> but only affecting RBAC debug mode. The problem is that `create_app` returns 
> a tuple, but the `cli` script expects to just receive the flask app back 
> without the appbuilder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-3125) Add monitoring on Task Instance creation rate



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3125.

   Resolution: Fixed
Fix Version/s: 2.0.0

> Add monitoring on Task Instance creation rate
> -
>
> Key: AIRFLOW-3125
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3125
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Mingye Xia
>Assignee: Mingye Xia
>Priority: Major
> Fix For: 2.0.0
>
>
> Monitoring on Task Instance creation rate can give us some visibility on how 
> much workload we are putting on Airflow. It can be used for resource 
> allocation in the long run (i.e. to determine when we should scale up 
> workers) and and debugging in scenarios like creation rate for certain types 
> of Task Instances spike.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] YingboWang edited a comment on issue #3798: [AIRFLOW-2951] Update dag_run table end_date when state change

YingboWang edited a comment on issue #3798: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: 
https://github.com/apache/incubator-airflow/pull/3798#issuecomment-426142027
 
 
   @ashb The experimental part in v1.10-test branch uses the 
DagRun.update_state() function defined in airflow/models.py so the 
experimental/mark_tasks.py is fully consistent with functionalities defined in 
models.py. We can safely ignore changing in mark_tasks.py and only change 
airflow/models.py. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3985: [AIRFLOW-3138] Use current data type for migrations

ashb commented on issue #3985: [AIRFLOW-3138] Use current data type for 
migrations
URL: 
https://github.com/apache/incubator-airflow/pull/3985#issuecomment-426199116
 
 
   @bolkedebruin This looks okay to me - do you have time for a quick check too.
   
   I think this is okay as our UtcDateTime type is the same column type under 
the hood. But thinking a bit more about it does doing this set the timezones 
incorrectly on the row. Hm.
   
   (Turns out there's no way to remove a review approval in github)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3138) Migration cc1e65623dc7 creates issues with postgres



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-3138:
---
Fix Version/s: 1.10.1

> Migration cc1e65623dc7 creates issues with postgres
> ---
>
> Key: AIRFLOW-3138
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3138
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: db
>Affects Versions: 1.10.0
>Reporter: M Noronha
>Priority: Major
> Fix For: 1.10.1
>
>
> When trying to upgrade from 1.8.0 to 1.10.0, migration 
> cc1e65623dc7_add_max_tries_column_to_task_instance.py results in error:
> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance' 
> expected to update 1 row(s); 0 were matched.
> That's due to a datatype mismatch between the database (TIMESTAMP before 
> migration 0e2a74e0fc9f) and the UtcDateTime.
> Changing 
> 55c54
> < execution_date = Column(UtcDateTime, primary_key=True)
> ---
> > execution_date = Column(sa.DateTime(), primary_key=True)
> Seems to be enough to work in postgres. This also works in sqlite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb commented on issue #3580: [AIRFLOW-1163] Add support for x-forwarded-* headers

ashb commented on issue #3580: [AIRFLOW-1163] Add support for x-forwarded-* 
headers
URL: 
https://github.com/apache/incubator-airflow/pull/3580#issuecomment-426196072
 
 
   Yeah, not merged in a way GitHub could see It (at the time we couldn't merge 
directly to Github) but 
https://github.com/apache/incubator-airflow/commit/a8dd6d86d512af2dcaf77270b18f2f6366bff5e6
 was the same commit on master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #3986: [AIRFLOW-3139] include parameters into log.info in SQL operators, if any

codecov-io commented on issue #3986: [AIRFLOW-3139] include parameters into 
log.info in SQL operators, if any
URL: 
https://github.com/apache/incubator-airflow/pull/3986#issuecomment-426187476
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=h1)
 Report
   > Merging 
[#3986](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/6aabdf16455cb6ab2cbe0f6b2cee7f28c852f368?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3986/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #3986  +/-   ##
   =
   + Coverage75.5%   75.5%   +<.01% 
   =
 Files 199 199  
 Lines   15945   15946   +1 
   =
   + Hits12039   12040   +1 
 Misses   39063906
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3986/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `79.83% <100%> (+0.16%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=footer).
 Last update 
[6aabdf1...4d0279a](https://codecov.io/gh/apache/incubator-airflow/pull/3986?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any

2018-10-02 Thread Xiaodong DENG (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-3139:
---
Description: 
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
log INFO.

But if parameters are used for the sql command, the parameters are not printed. 
This makes the logging less useful.

 

*Example:*

!Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!

 

  was:
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
log INFO.

But if parameters are used for the sql command, the parameters are not printed. 
This makes the printing useless.

 

*Example:*

!Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!

 


> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
> log INFO.
> But if parameters are used for the sql command, the parameters are not 
> printed. This makes the logging less useful.
>  
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] itscaro removed a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators

itscaro removed a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push 
flag for contrib's operators
URL: 
https://github.com/apache/incubator-airflow/pull/3981#issuecomment-426179746
 
 
   I got two errors and I could not find where they are coming from, if anyone 
can have a look, it would be great. Curiously the labels is set to be None.
   
   ```==
   23) FAIL: test_exec 
(tests.contrib.operators.test_dataflow_operator.DataFlowPythonOperatorTest)
   --
  Traceback (most recent call last):
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 1305 in patched
 return func(*args, **keywargs)
   tests/contrib/operators/test_dataflow_operator.py line 115 in test_exec
 PY_OPTIONS)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: 
start_python_dataflow('test-dataflow-pipeline', {'labels': {'airflow-version': 
'v2-0-0-dev0-incubating', 'foo': 'bar'}, 'project': 'test', 'output': 
'gs://test/output', 'staging_location': 'gs://test/staging'}, , ['-m'])
  Actual call: start_python_dataflow('test-dataflow-pipeline', {None: 
'gs://test/output'}, , 
['-m'])
   ==
   24) FAIL: testSuccessfulRun 
(tests.contrib.operators.test_mlengine_operator_utils.CreateEvaluateOpsTest)
   --
  Traceback (most recent call last):
   tests/contrib/operators/test_mlengine_operator_utils.py line 125 in 
testSuccessfulRun
 ['-m'])
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: start_python_dataflow('{{task.task_id}}', 
{'metric_keys': 'err', 'prediction_path': 'gs://legal-bucket/fake-output-path', 
'metric_fn_encoded': 
'gAJjZGlsbC5fZGlsbApfY3JlYXRlX2Z1bmN0aW9uCnEAKGNkaWxsLl9kaWxsCl9sb2FkX3R5cGUKcQFVCENvZGVUeXBlcQKFcQNScQQoSwFLAUsBSlNgAQBVBGQCAFNxBU5HP7mZmZmZmZpHP7mZmZmZmZqFcQaHcQcpVQF4cQiFcQlVPC9hcHAvdGVzdHMvY29udHJpYi9vcGVyYXRvcnMvdGVzdF9tbGVuZ2luZV9vcGVyYXRvcl91dGlscy5weXEKVQg8bGFtYmRhPnELS0lVAHEMKSl0cQ1ScQ59cQ9oC05OfXEQdHERUnESLg==',
 'labels': {'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
  Actual call: start_python_dataflow('{{task.task_id}}', {None: 
{'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators

itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push 
flag for contrib's operators
URL: 
https://github.com/apache/incubator-airflow/pull/3981#issuecomment-426179746
 
 
   I got two errors and I could not find where they are coming from, if anyone 
can have a look, it would be great. Curiously the labels is set to be None.
   
   ```==
   23) FAIL: test_exec 
(tests.contrib.operators.test_dataflow_operator.DataFlowPythonOperatorTest)
   --
  Traceback (most recent call last):
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 1305 in patched
 return func(*args, **keywargs)
   tests/contrib/operators/test_dataflow_operator.py line 115 in test_exec
 PY_OPTIONS)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: 
start_python_dataflow('test-dataflow-pipeline', {'labels': {'airflow-version': 
'v2-0-0-dev0-incubating', 'foo': 'bar'}, 'project': 'test', 'output': 
'gs://test/output', 'staging_location': 'gs://test/staging'}, , ['-m'])
  Actual call: start_python_dataflow('test-dataflow-pipeline', {None: 
'gs://test/output'}, , 
['-m'])
   ==
   24) FAIL: testSuccessfulRun 
(tests.contrib.operators.test_mlengine_operator_utils.CreateEvaluateOpsTest)
   --
  Traceback (most recent call last):
   tests/contrib/operators/test_mlengine_operator_utils.py line 125 in 
testSuccessfulRun
 ['-m'])
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: start_python_dataflow('{{task.task_id}}', 
{'metric_keys': 'err', 'prediction_path': 'gs://legal-bucket/fake-output-path', 
'metric_fn_encoded': 
'gAJjZGlsbC5fZGlsbApfY3JlYXRlX2Z1bmN0aW9uCnEAKGNkaWxsLl9kaWxsCl9sb2FkX3R5cGUKcQFVCENvZGVUeXBlcQKFcQNScQQoSwFLAUsBSlNgAQBVBGQCAFNxBU5HP7mZmZmZmZpHP7mZmZmZmZqFcQaHcQcpVQF4cQiFcQlVPC9hcHAvdGVzdHMvY29udHJpYi9vcGVyYXRvcnMvdGVzdF9tbGVuZ2luZV9vcGVyYXRvcl91dGlscy5weXEKVQg8bGFtYmRhPnELS0lVAHEMKSl0cQ1ScQ59cQ9oC05OfXEQdHERUnESLg==',
 'labels': {'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
  Actual call: start_python_dataflow('{{task.task_id}}', {None: 
{'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] itscaro commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators

itscaro commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for 
contrib's operators
URL: 
https://github.com/apache/incubator-airflow/pull/3981#issuecomment-426179746
 
 
   I got two errors and I could not find where they are coming from, if anyone 
can have a look, it would be great.
   
   ```==
   23) FAIL: test_exec 
(tests.contrib.operators.test_dataflow_operator.DataFlowPythonOperatorTest)
   --
  Traceback (most recent call last):
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 1305 in patched
 return func(*args, **keywargs)
   tests/contrib/operators/test_dataflow_operator.py line 115 in test_exec
 PY_OPTIONS)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: 
start_python_dataflow('test-dataflow-pipeline', {'labels': {'airflow-version': 
'v2-0-0-dev0-incubating', 'foo': 'bar'}, 'project': 'test', 'output': 
'gs://test/output', 'staging_location': 'gs://test/staging'}, , ['-m'])
  Actual call: start_python_dataflow('test-dataflow-pipeline', {None: 
'gs://test/output'}, , 
['-m'])
   ==
   24) FAIL: testSuccessfulRun 
(tests.contrib.operators.test_mlengine_operator_utils.CreateEvaluateOpsTest)
   --
  Traceback (most recent call last):
   tests/contrib/operators/test_mlengine_operator_utils.py line 125 in 
testSuccessfulRun
 ['-m'])
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 948 in assert_called_once_with
 return self.assert_called_with(*args, **kwargs)
   
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py 
line 937 in assert_called_with
 six.raise_from(AssertionError(_error_message(cause)), cause)
   .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py 
line 737 in raise_from
 raise value
  AssertionError: Expected call: start_python_dataflow('{{task.task_id}}', 
{'metric_keys': 'err', 'prediction_path': 'gs://legal-bucket/fake-output-path', 
'metric_fn_encoded': 
'gAJjZGlsbC5fZGlsbApfY3JlYXRlX2Z1bmN0aW9uCnEAKGNkaWxsLl9kaWxsCl9sb2FkX3R5cGUKcQFVCENvZGVUeXBlcQKFcQNScQQoSwFLAUsBSlNgAQBVBGQCAFNxBU5HP7mZmZmZmZpHP7mZmZmZmZqFcQaHcQcpVQF4cQiFcQlVPC9hcHAvdGVzdHMvY29udHJpYi9vcGVyYXRvcnMvdGVzdF9tbGVuZ2luZV9vcGVyYXRvcl91dGlscy5weXEKVQg8bGFtYmRhPnELS0lVAHEMKSl0cQ1ScQ59cQ9oC05OfXEQdHERUnESLg==',
 'labels': {'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
  Actual call: start_python_dataflow('{{task.task_id}}', {None: 
{'airflow-version': 'v2-0-0-dev0-incubating'}}, 
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635108#comment-16635108
 ] 

ASF GitHub Bot commented on AIRFLOW-3139:
-

XD-DENG opened a new pull request #3986: [AIRFLOW-3139] include parameters into 
log.info in SQL operators, if any
URL: https://github.com/apache/incubator-airflow/pull/3986
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3139
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   For all SQL-operators based on `DbApiHook`, sql command itself is printed 
into `log.info`. But if parameters are used for the sql command, the parameters 
would not be included in the printing. This makes the log less useful.
   
   This commit ensures that the parameters are also printed into the 
`log.info`, if any.
   
   **Before this commit**
   https://user-images.githubusercontent.com/11539188/46335270-7f019800-c659-11e8-89ff-09fc3ed7850e.png;>
   
   **After this commit**
   (if no parameters is given for the sql command, there will be no change)
   https://user-images.githubusercontent.com/11539188/46335305-950f5880-c659-11e8-985e-b0fd7ea85c53.png;>
   
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
> log INFO.
> But if parameters are used for the sql command, the parameters are not 
> printed. This makes the printing useless.
>  
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG opened a new pull request #3986: [AIRFLOW-3139] include parameters into log.info in SQL operators, if any

XD-DENG opened a new pull request #3986: [AIRFLOW-3139] include parameters into
log.info in SQL operators, if any
URL: https://github.com/apache/incubator-airflow/pull/3986

Make sure you have checked _all_ steps below.

### Jira

- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3139
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.

### Description

- [x] Here are some details about my PR, including screenshots of any UI
changes:

For all SQL-operators based on `DbApiHook`, sql command itself is printed
into `log.info`. But if parameters are used for the sql command, the parameters
would not be included in the printing. This makes the log less useful.

This commit ensures that the parameters are also printed into the
`log.info`, if any.

**Before this commit**
https://user-images.githubusercontent.com/11539188/46335270-7f019800-c659-11e8-89ff-09fc3ed7850e.png;>

**After this commit**
(if no parameters is given for the sql command, there will be no change)
https://user-images.githubusercontent.com/11539188/46335305-950f5880-c659-11e8-985e-b0fd7ea85c53.png;>

### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.

### Code Quality

- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3123) Allow nested use of DAG as a context manager

2018-10-02 Thread Newton Le (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Newton Le resolved AIRFLOW-3123.

Resolution: Fixed

> Allow nested use of  DAG as a context manager
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
>   op1 = Operator()
>   with dag:
> op2 = Operator()
>   op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the 
> nested context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3123) Allow nested use of DAG as a context manager



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635103#comment-16635103
 ] 

Iuliia Volkova commented on AIRFLOW-3123:
-

[~newtonle], [~ashb], please don't forget to close the issue, PR was merged. 

> Allow nested use of  DAG as a context manager
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
>   op1 = Operator()
>   with dag:
> op2 = Operator()
>   op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the 
> nested context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3124) Broken webserver debug mode (RBAC)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635101#comment-16635101
 ] 

Iuliia Volkova commented on AIRFLOW-3124:
-

PR was merged. Could we close the issue? [~ajkosel] [~TaoFeng]

> Broken webserver debug mode (RBAC)
> --
>
> Key: AIRFLOW-3124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3124
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp, webserver
>Reporter: Aaron Kosel
>Assignee: Aaron Kosel
>Priority: Minor
>
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 7, in 
> exec(compile(f.read(), __file__, 'exec'))
> File "/airflow/airflow/bin/airflow", line 32, in 
> args.func(args)
> File "/airflow/airflow/utils/cli.py", line 74, in wrapper
> return f(*args, **kwargs)
> File "/airflow/airflow/bin/cli.py", line 875, in webserver
> app.run(debug=True, port=args.port, host=args.hostname,
> AttributeError: 'tuple' object has no attribute 'run'
> {code}
> Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204, 
> but only affecting RBAC debug mode. The problem is that `create_app` returns 
> a tuple, but the `cli` script expects to just receive the flask app back 
> without the appbuilder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3125) Add monitoring on Task Instance creation rate



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635100#comment-16635100
 ] 

Iuliia Volkova commented on AIRFLOW-3125:
-

[~xiamingye], don't forget to close JIRA issue :) your PR already merged!
[~ashb]

> Add monitoring on Task Instance creation rate
> -
>
> Key: AIRFLOW-3125
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3125
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Mingye Xia
>Assignee: Mingye Xia
>Priority: Major
>
> Monitoring on Task Instance creation rate can give us some visibility on how 
> much workload we are putting on Airflow. It can be used for resource 
> allocation in the long run (i.e. to determine when we should scale up 
> workers) and and debugging in scenarios like creation rate for certain types 
> of Task Instances spike.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3129) Improve test coverage