date:20190425

raviagarwalunravel commented on issue #5118: [AIRFLOW-4315] Add monitoring 
API's to airflow
URL: https://github.com/apache/airflow/pull/5118#issuecomment-486926276
 
 
   > Found out you don't change code in CI, so I think maybe a flaky test, you 
should using git force-push to restart CI
   
   Thanks, I am re-organizing my commits, I will do a force push after that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on issue #5118: [AIRFLOW-4315] Add monitoring API's to airflow

zhongjiajie commented on issue #5118: [AIRFLOW-4315] Add monitoring API's to 
airflow
URL: https://github.com/apache/airflow/pull/5118#issuecomment-486924231
 
 
   @raviagarwalunravel FYI
   Error log like below
   ```
   Traceback (most recent call last):
 File "/app/.tox/py35-backend_sqlite-env_docker/bin/airflow", line 7, in 

   exec(compile(f.read(), __file__, 'exec'))
 File "/app/airflow/bin/airflow", line 32, in 
   args.func(args)
 File "/app/airflow/utils/cli.py", line 73, in wrapper
   return f(*args, **kwargs)
 File "/app/airflow/bin/cli.py", line 522, in run
   _run(args, dag, ti)
 File "/app/airflow/bin/cli.py", line 435, in _run
   run_job.run()
 File "/app/airflow/jobs.py", line 206, in run
   self._execute()
 File "/app/airflow/jobs.py", line 2639, in _execute
   self.on_kill()
 File "/app/airflow/jobs.py", line 2642, in on_kill
   self.task_runner.terminate()
 File "/app/airflow/task/task_runner/standard_task_runner.py", line 41, in 
terminate
   reap_process_group(self.process.pid, self.log)
 File "/app/airflow/utils/helpers.py", line 276, in reap_process_group
   os.killpg(os.getpgid(pid), sig)
   PermissionError: [Errno 1] Operation not permitted
   userdel: airflow_test_user mail spool (/var/mail/airflow_test_user) not found
   ```
   Found out you don't change code in CI, so I think maybe a flaky test, you 
should using git force-push to restart CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

codecov-io edited a comment on issue #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182#issuecomment-486920809
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=h1) 
Report
   > Merging 
[#5182](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5182   +/-   ##
   ===
 Coverage   78.47%   78.47%   
   ===
 Files 467  467   
 Lines   2978329783   
   ===
 Hits2337323373   
 Misses   6410 6410
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=footer). 
Last update 
[cb9b468...27deb1b](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

codecov-io commented on issue #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182#issuecomment-486920808
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=h1) 
Report
   > Merging 
[#5182](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5182   +/-   ##
   ===
 Coverage   78.47%   78.47%   
   ===
 Files 467  467   
 Lines   2978329783   
   ===
 Hits2337323373   
 Misses   6410 6410
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=footer). 
Last update 
[cb9b468...27deb1b](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

codecov-io commented on issue #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182#issuecomment-486920809
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=h1) 
Report
   > Merging 
[#5182](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5182   +/-   ##
   ===
 Coverage   78.47%   78.47%   
   ===
 Files 467  467   
 Lines   2978329783   
   ===
 Hits2337323373   
 Misses   6410 6410
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=footer). 
Last update 
[cb9b468...27deb1b](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

codecov-io edited a comment on issue #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182#issuecomment-486920809
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=h1) 
Report
   > Merging 
[#5182](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5182/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5182   +/-   ##
   ===
 Coverage   78.47%   78.47%   
   ===
 Files 467  467   
 Lines   2978329783   
   ===
 Hits2337323373   
 Misses   6410 6410
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=footer). 
Last update 
[cb9b468...27deb1b](https://codecov.io/gh/apache/airflow/pull/5182?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix

codecov-io edited a comment on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix
URL: https://github.com/apache/airflow/pull/5190#issuecomment-486915771
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=h1) 
Report
   > Merging 
[#5190](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `80%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5190/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5190  +/-   ##
   ==
   - Coverage   78.47%   78.47%   -0.01% 
   ==
 Files 467  467  
 Lines   2978329788   +5 
   ==
   + Hits2337323376   +3 
   - Misses   6410 6412   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `83.72% <80%> (-0.16%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.42% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=footer). 
Last update 
[cb9b468...8052ab0](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix

codecov-io edited a comment on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix
URL: https://github.com/apache/airflow/pull/5190#issuecomment-486915771
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=h1) 
Report
   > Merging 
[#5190](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `80%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5190/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5190  +/-   ##
   ==
   - Coverage   78.47%   78.47%   -0.01% 
   ==
 Files 467  467  
 Lines   2978329788   +5 
   ==
   + Hits2337323376   +3 
   - Misses   6410 6412   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `83.72% <80%> (-0.16%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.42% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=footer). 
Last update 
[cb9b468...8052ab0](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix

codecov-io commented on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix
URL: https://github.com/apache/airflow/pull/5190#issuecomment-486915771
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=h1) 
Report
   > Merging 
[#5190](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/cb9b4682fa7f142ffe06837f7856cb88b75a3aa4?src=pr=desc)
 will **decrease** coverage by `0.67%`.
   > The diff coverage is `80%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5190/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #5190  +/-   ##
   =
   - Coverage   78.47%   77.8%   -0.68% 
   =
 Files 467 467  
 Lines   29783   29788   +5 
   =
   - Hits23373   23176 -197 
   - Misses   64106612 +202
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `83.72% <80%> (-0.16%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `38.46% <0%> (-38.47%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `75% <0%> (-5.96%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `85.34% <0%> (-3.45%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `92.64% <0%> (-2.95%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `91.66% <0%> (-2.78%)` | :arrow_down: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/5190/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=footer). 
Last update 
[cb9b468...8052ab0](https://codecov.io/gh/apache/airflow/pull/5190?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4361) Fix flaky test_integration_run_dag_with_scheduler_failure



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826616#comment-16826616
 ] 

ASF GitHub Bot commented on AIRFLOW-4361:
-

milton0825 commented on pull request #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix flaky test_integration_run_dag_with_scheduler_failure
> -
>
> Key: AIRFLOW-4361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4361
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> test_integration_run_dag_with_scheduler_failure often fails with
> {code}
> ^[[33m   ^[[33m^[[1m^[[33mConnectionError^[[0m^[[0m^[[33m: 
> ^[[0m^[[33mHTTPConnectionPool(host='10.20.3.19', port=  30809): Max 
> retries exceeded with url: 
> /api/experimental/dags/example_kubernetes_executor_config/paused/false (Ca
>   used by NewConnectionError(' 0x7fb80986f208>: Failed to establish a n  ew connection: [Errno 111] 
> Connection refused',))^[[0m^M
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4361) Fix flaky test_integration_run_dag_with_scheduler_failure



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826617#comment-16826617
 ] 

ASF GitHub Bot commented on AIRFLOW-4361:
-

milton0825 commented on pull request #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4361
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Block until airflow webserver is up and running.
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [X] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix flaky test_integration_run_dag_with_scheduler_failure
> -
>
> Key: AIRFLOW-4361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4361
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> test_integration_run_dag_with_scheduler_failure often fails with
> {code}
> ^[[33m   ^[[33m^[[1m^[[33mConnectionError^[[0m^[[0m^[[33m: 
> ^[[0m^[[33mHTTPConnectionPool(host='10.20.3.19', port=  30809): Max 
> retries exceeded with url: 
> /api/experimental/dags/example_kubernetes_executor_config/paused/false (Ca
>   used by NewConnectionError(' 0x7fb80986f208>: Failed to establish a n  ew connection: [Errno 111] 
> Connection refused',))^[[0m^M
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] zhongjiajie commented on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix

zhongjiajie commented on issue #5190: [AIRFLOW-XXX]pymysql pendulum fix
URL: https://github.com/apache/airflow/pull/5190#issuecomment-486914821
 
 
   Please add jira ticket due to you change code. @nazo 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 opened a new pull request #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

milton0825 opened a new pull request #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4361
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Block until airflow webserver is up and running.
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [X] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 closed pull request #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure

milton0825 closed pull request #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nazo opened a new pull request #5190: [AIRFLOW-XXX]pymysql pendulum fix

nazo opened a new pull request #5190: [AIRFLOW-XXX]pymysql pendulum fix
URL: https://github.com/apache/airflow/pull/5190
 
 
   ### Jira
   
   ### Description
   
   see
   https://github.com/sdispater/pendulum/issues/267
   https://github.com/sdispater/pendulum/tree/1.x#limitations
   
   ### Tests
   
   ### Commits
   
   
   ### Documentation
   
   
   ### Code Quality
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor

codecov-io edited a comment on issue #5166: [AIRFLOW-4397] Add 
GCSUploadSessionCompleteSensor
URL: https://github.com/apache/airflow/pull/5166#issuecomment-486420734
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=h1) 
Report
   > Merging 
[#5166](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/de1795c8f1e65a5f4a848e88ef3848ba51aab449?src=pr=desc)
 will **increase** coverage by `0.12%`.
   > The diff coverage is `93.61%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5166/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5166  +/-   ##
   ==
   + Coverage   78.44%   78.57%   +0.12% 
   ==
 Files 466  467   +1 
 Lines   2973529830  +95 
   ==
   + Hits2332723438 +111 
   + Misses   6408 6392  -16
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/sensors/gcs\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NzX3NlbnNvci5weQ==)
 | `69.47% <93.61%> (+69.47%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/openfaas\_hook.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL29wZW5mYWFzX2hvb2sucHk=)
 | `94.11% <0%> (-0.12%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `93.93% <0%> (-0.1%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `95.58% <0%> (-0.07%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.27% <0%> (-0.06%)` | :arrow_down: |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <0%> (ø)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzLnB5)
 | `97.87% <0%> (ø)` | :arrow_up: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `91.81% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/qubole\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfY2hlY2tfb3BlcmF0b3IucHk=)
 | `71.76% <0%> (ø)` | :arrow_up: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=footer). 
Last update 
[de1795c...6b85cd6](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor

2019-04-25 Thread ASF subversion and git services (JIRA)

codecov-io edited a comment on issue #5166: [AIRFLOW-4397] Add 
GCSUploadSessionCompleteSensor
URL: https://github.com/apache/airflow/pull/5166#issuecomment-486420734
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=h1) 
Report
   > Merging 
[#5166](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/de1795c8f1e65a5f4a848e88ef3848ba51aab449?src=pr=desc)
 will **increase** coverage by `0.12%`.
   > The diff coverage is `93.61%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5166/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5166  +/-   ##
   ==
   + Coverage   78.44%   78.57%   +0.12% 
   ==
 Files 466  467   +1 
 Lines   2973529830  +95 
   ==
   + Hits2332723438 +111 
   + Misses   6408 6392  -16
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/sensors/gcs\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NzX3NlbnNvci5weQ==)
 | `69.47% <93.61%> (+69.47%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/openfaas\_hook.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL29wZW5mYWFzX2hvb2sucHk=)
 | `94.11% <0%> (-0.12%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `93.93% <0%> (-0.1%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `95.58% <0%> (-0.07%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.27% <0%> (-0.06%)` | :arrow_down: |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <0%> (ø)` | :arrow_up: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzLnB5)
 | `97.87% <0%> (ø)` | :arrow_up: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `91.81% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/qubole\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfY2hlY2tfb3BlcmF0b3IucHk=)
 | `71.76% <0%> (ø)` | :arrow_up: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/5166/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=footer). 
Last update 
[de1795c...6b85cd6](https://codecov.io/gh/apache/airflow/pull/5166?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4409) UI duration view can be broken by task faile null duration column



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826530#comment-16826530
 ] 

ASF subversion and git services commented on AIRFLOW-4409:
--

Commit cb9b4682fa7f142ffe06837f7856cb88b75a3aa4 in airflow's branch 
refs/heads/master from Yingbo Wang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=cb9b468 ]

[AIRFLOW-4409] Prevent task duration break by null value (#5178)



> UI duration view can be broken by task faile null duration column
> -
>
> Key: AIRFLOW-4409
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4409
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 1.10.0
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Minor
> Attachments: Screen Shot 2019-04-24 at 1.54.53 PM.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Airflow DAG UI has a view "task duration". Due to our recent migration. There 
> are some records in the task_fail table which has null duration (caused by 
> missing start date). This is affecting the UI view of task duration  !Screen 
> Shot 2019-04-24 at 1.54.53 PM.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-4409) UI duration view can be broken by task faile null duration column

2019-04-25 Thread Tao Feng (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng resolved AIRFLOW-4409.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> UI duration view can be broken by task faile null duration column
> -
>
> Key: AIRFLOW-4409
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4409
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 1.10.0
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Screen Shot 2019-04-24 at 1.54.53 PM.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Airflow DAG UI has a view "task duration". Due to our recent migration. There 
> are some records in the task_fail table which has null duration (caused by 
> missing start date). This is affecting the UI view of task duration  !Screen 
> Shot 2019-04-24 at 1.54.53 PM.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4409) UI duration view can be broken by task faile null duration column



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826529#comment-16826529
 ] 

ASF GitHub Bot commented on AIRFLOW-4409:
-

feng-tao commented on pull request #5178: [AIRFLOW-4409] Prevent task duration 
view broken by null duration
URL: https://github.com/apache/airflow/pull/5178
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> UI duration view can be broken by task faile null duration column
> -
>
> Key: AIRFLOW-4409
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4409
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 1.10.0
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Minor
> Attachments: Screen Shot 2019-04-24 at 1.54.53 PM.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Airflow DAG UI has a view "task duration". Due to our recent migration. There 
> are some records in the task_fail table which has null duration (caused by 
> missing start date). This is affecting the UI view of task duration  !Screen 
> Shot 2019-04-24 at 1.54.53 PM.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao merged pull request #5178: [AIRFLOW-4409] Prevent task duration view broken by null duration

feng-tao merged pull request #5178: [AIRFLOW-4409] Prevent task duration view 
broken by null duration
URL: https://github.com/apache/airflow/pull/5178
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5178: [AIRFLOW-4409] Prevent task duration view broken by null duration

feng-tao commented on issue #5178: [AIRFLOW-4409] Prevent task duration view 
broken by null duration
URL: https://github.com/apache/airflow/pull/5178#issuecomment-486870974
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jaketf commented on issue #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor

jaketf commented on issue #5166: [AIRFLOW-4397] Add 
GCSUploadSessionCompleteSensor
URL: https://github.com/apache/airflow/pull/5166#issuecomment-486863582
 
 
   @potiuk this is ready for review again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] fmao opened a new pull request #5189: [AIRFLOW-4415] Speedup status propagation

fmao opened a new pull request #5189: [AIRFLOW-4415] Speedup status propagation
URL: https://github.com/apache/airflow/pull/5189

Make sure you have checked _all_ steps below.

### Jira

My PR addresses the following Airflow JIRA issues and references them in the
PR title.
https://issues.apache.org/jira/browse/AIRFLOW-4415

references:
https://issues.apache.org/jira/browse/AIRFLOW-719
https://issues.apache.org/jira/browse/AIRFLOW-982
https://issues.apache.org/jira/browse/AIRFLOW-983

The temp fix was remove after 1.8.0 due to
https://github.com/apache/airflow/pull/2195

in the latest branch, a new trigger rule is added.
https://github.com/apache/airflow/pull/4182

### Description

Issue: skip status stop propagation to down streams and get randomly stopped
with the dag status marked as failed.
The issue is located in the version 1.8.1.
In version 1.8.0 there is a temp fix but removed after this version.

https://github.com/apache/airflow/commit/4077c6de297566a4c598065867a9a27324ae6eb1

https://github.com/apache/airflow/commit/92965e8275c6f2ec2282ad46c09950bab10c1cb2

root casue:
In a loop, the scheduler evaluates each dag and all its task dependcies
around by around.
Each round evaluation happens twice in the context of flag_upstream_failed
= false and =true.

The dag run update method mark the dag run deadlocked which stops the dag
and all its tasks from be processed furture.
https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4184
It is due to in no_dependencies_met. All_sccucess trigger rule misses
skipped status check and marks the task as failed when upstream only has
skipped tasks.
https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4152

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L165

Each dag update will checks all its task deps and sent ready tasks to run
in the context of flag_upstream_failed=false (defalt)
https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4156
which wont handle skip status propagation.

After dag update, dag will checks all its task deps and sent ready tasks
to run in the context of flag_upstream_failed=true
https://github.com/apache/airflow/blob/1.8.1/airflow/jobs.py#L904
which handles skip status propogration.

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L138

Two potential causes that will trigger dag update detect a deadlock.
The skip status proprogatation rely on detected skipped upstreams (which
happens asyncly by other nodes writing skipped status to db).
If the tasks been evaluated are not following topoloy order(random order)
by priority_weigth. It requried multipe loop rounds to propogate skip statue to
all downsteam tasks.
Depending on how close the topoloy order the tasks fetched, the
proprogation may go further or shorter.

The deadlock detetion can be avoid only the following conditions happen
at the same time:
1. the skip task (shortcurit operation async process) update db with skip
task status, right after dag update (flag_upstream_failed=false )before dag
task checks(flag_upstream_failed=true) in scheduler process.
2. dag checks(flag_upstream_failed=true) have all tasks fetched/evaluated
in the topology order that skip status can propogate in one evaluations round.

Fix approaches:

1) Mark the ALL_sucess trigger rule : num_failures = upstream - successes -
skipped.
It will prevent the deadlock detectoin from being triggered. If the tasks
are not ordered, mulitple rounds are required and will eventually mark all of
the tasked as skipped. Or add an additional trigger rule. That is
https://github.com/apache/airflow/pull/4182 and make it as the default trigger
rule.

2) Ordered tasks by topopy order to speed speedup skip status propogation in
one round of evaluation.
https://github.com/apache/airflow/blob/1.8.1/airflow/jobs.py#L893
tis = sorted(tis, key=lambda x: x.priority_weight, reverse=True)

### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ]

[jira] [Commented] (AIRFLOW-4415) skip status stops propagation randomly.

[
https://issues.apache.org/jira/browse/AIRFLOW-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826462#comment-16826462
]

ASF GitHub Bot commented on AIRFLOW-4415:
-

fmao commented on pull request #5189: [AIRFLOW-4415] Speedup status propagation
URL: https://github.com/apache/airflow/pull/5189

Make sure you have checked _all_ steps below.

### Jira

My PR addresses the following Airflow JIRA issues and references them in the
PR title.
https://issues.apache.org/jira/browse/AIRFLOW-4415

references:
https://issues.apache.org/jira/browse/AIRFLOW-719
https://issues.apache.org/jira/browse/AIRFLOW-982
https://issues.apache.org/jira/browse/AIRFLOW-983

The temp fix was remove after 1.8.0 due to
https://github.com/apache/airflow/pull/2195

in the latest branch, a new trigger rule is added.
https://github.com/apache/airflow/pull/4182

### Description

https://github.com/apache/airflow/commit/4077c6de297566a4c598065867a9a27324ae6eb1

https://github.com/apache/airflow/commit/92965e8275c6f2ec2282ad46c09950bab10c1cb2

root casue:
In a loop, the scheduler evaluates each dag and all its task dependcies
around by around.
Each round evaluation happens twice in the context of flag_upstream_failed
= false and =true.

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L165

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L138

Fix approaches:

### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

[GitHub] [airflow] fmao closed pull request #5188: status propagation

fmao closed pull request #5188: status propagation
URL: https://github.com/apache/airflow/pull/5188
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4415) skip status stops propagation randomly.

2019-04-25 Thread Feng Mao (JIRA)

Feng Mao created AIRFLOW-4415:
-

Summary: skip status stops propagation randomly.
Key: AIRFLOW-4415
URL: https://issues.apache.org/jira/browse/AIRFLOW-4415
Project: Apache Airflow
Issue Type: Improvement
Components: scheduler
Affects Versions: 1.8.1
Reporter: Feng Mao
Fix For: 1.10.4

Issue: skip status stop propogation to down streams and get randomly stopped
with the dag status marked as failed.

The issue is located in the version 1.8.1.

In version 1.8.0 there is a temp fix but removed after this version.

https://github.com/apache/airflow/commit/4077c6de297566a4c598065867a9a27324ae6eb1

https://github.com/apache/airflow/commit/92965e8275c6f2ec2282ad46c09950bab10c1cb2

root casue:

In a loop, the scheduler evaluate each dag and all its task dependcies around
by around.

Each round evaluation happens twice in the context of flag_upstream_failed =
false and true.

The dag run update method mark the dag run deadlocked which stops the dag and
all its tasks from be processed furture.

https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4184

It is due to in no_dependencies_met. All_sccucess trigger rule misses
skipped status check and marks the task as failed when upstream only has
skipped tasks.

https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4152

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L165

Each dag update will checks all its task deps and sent ready tasks to run in
the context of flag_upstream_failed=false (defalt)

https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4156 which
wont handle skip status propogation.

After dag update, dag will checks all its task deps and sent ready tasks to
run in the context of flag_upstream_failed=true

https://github.com/apache/airflow/blob/1.8.1/airflow/jobs.py#L904

which handles skip status propogration.

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L138

Two potential causes that will trigger dag update detect a deadlock.

The skip status proprogatation rely on detected skipped upstreams (which
happens asyncly by other nodes writing to db).

If the tasks been evaluated are not following topoloy order(random order) by
priority_weigth. It requried multipe loop rounds to propogate skip statue to
all downsteam tasks.

Depending on how close the topoloy order the tasks fetched, the proprogation
may go further or shorter.

The deadlock detetion can be avoid only the following conditions happen at
the same time:

1. the skip task (shortcurit operation async process) update db with skipped
task status, right after dag update (flag_upstream_failed=false )before dag
task checks(flag_upstream_failed=true) in scheduler process.

2. dag checks(flag_upstream_failed=true) have all tasks fectch/evaluated in
the topology order that skip status can propogate in one evaluations round.

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] fmao opened a new pull request #5188: status propagation

fmao opened a new pull request #5188: status propagation
URL: https://github.com/apache/airflow/pull/5188

Make sure you have checked _all_ steps below.

### Jira

My PR addresses the following Airflow JIRA issues and references them in the
PR title.
https://issues.apache.org/jira/browse/AIRFLOW-719
https://issues.apache.org/jira/browse/AIRFLOW-982
https://issues.apache.org/jira/browse/AIRFLOW-983

The temp fix was remove after 1.8.0 due to
https://github.com/apache/airflow/pull/2195

in the latest branch, a new trigger rule is added.
https://github.com/apache/airflow/pull/4182

### Description

https://github.com/apache/airflow/commit/4077c6de297566a4c598065867a9a27324ae6eb1

https://github.com/apache/airflow/commit/92965e8275c6f2ec2282ad46c09950bab10c1cb2

root casue:
In a loop, the scheduler evaluates each dag and all its task dependcies
around by around.
Each round evaluation happens twice in the context of flag_upstream_failed
= false and =true.

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L165

https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L138

Fix approaches:

### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All

[jira] [Commented] (AIRFLOW-4414) AWSAthenaOperator does not push QueryExecutionID to XCom



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826435#comment-16826435
 ] 

ASF GitHub Bot commented on AIRFLOW-4414:
-

mrshu commented on pull request #5187: [AIRFLOW-4414] AWSAthenaOperator: Push 
QueryExecutionID to XCom
URL: https://github.com/apache/airflow/pull/5187
 
 
   
   
   Signed-off-by: mr.Shu 
   -
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-4414) issues and references 
them in the PR title.
 - https://issues.apache.org/jira/browse/AIRFLOW-4414
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   * Currently it is not possible to make use of QueryExecutionID (the
 unique identifier of the query submitted to Athena) in the tasks that
 follow.
   
   * This commit changes the situation by making sure QueryExecutionID gets
 pushed to XCom when BaseOperator.do_xcom_push is set to True.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   There currently does not seem to exist a simple way of testing XCom pushing 
of Operators.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> AWSAthenaOperator does not push QueryExecutionID to XCom
> 
>
> Key: AIRFLOW-4414
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4414
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Reporter: Marek Suppa
>Assignee: Marek Suppa
>Priority: Minor
>
> AWSAthenaOperator should make it easy to push QueryExecutionID to XCom which 
> it currently does not. The best way of achieving that would currently most 
> probably be
> {code}
> class MyAWSAthenaOperator(AWSAthenaOperator):
> def execute(self, context):
> super(MyAWSAthenaOperator, self).execute(context)
> # just so that this gets `xcom_push`(ed)
> return self.query_execution_id
> {code}
> The fix should be as simple as adding {code}return 
> self.query_execution_id{code} at the end of {{execute}} function.
> I will add that as part of this task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] mrshu opened a new pull request #5187: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom

mrshu opened a new pull request #5187: [AIRFLOW-4414] AWSAthenaOperator: Push 
QueryExecutionID to XCom
URL: https://github.com/apache/airflow/pull/5187
 
 
   
   
   Signed-off-by: mr.Shu 
   -
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-4414) issues and references 
them in the PR title.
 - https://issues.apache.org/jira/browse/AIRFLOW-4414
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   * Currently it is not possible to make use of QueryExecutionID (the
 unique identifier of the query submitted to Athena) in the tasks that
 follow.
   
   * This commit changes the situation by making sure QueryExecutionID gets
 pushed to XCom when BaseOperator.do_xcom_push is set to True.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   There currently does not seem to exist a simple way of testing XCom pushing 
of Operators.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4414) AWSAthenaOperator does not push QueryExecutionID to XCom

2019-04-25 Thread Marek Suppa (JIRA)

Marek Suppa created AIRFLOW-4414:


 Summary: AWSAthenaOperator does not push QueryExecutionID to XCom
 Key: AIRFLOW-4414
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4414
 Project: Apache Airflow
  Issue Type: Improvement
  Components: aws
Reporter: Marek Suppa
Assignee: Marek Suppa


AWSAthenaOperator should make it easy to push QueryExecutionID to XCom which it 
currently does not. The best way of achieving that would currently most 
probably be

{code}
class MyAWSAthenaOperator(AWSAthenaOperator):
def execute(self, context):
super(MyAWSAthenaOperator, self).execute(context)
# just so that this gets `xcom_push`(ed)
return self.query_execution_id
{code}

The fix should be as simple as adding {code}return 
self.query_execution_id{code} at the end of {{execute}} function.

I will add that as part of this task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] bolkedebruin commented on a change in pull request #5048: [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler

bolkedebruin commented on a change in pull request #5048: [AIRFLOW-3370] Add 
stdout output options to Elasticsearch task log handler
URL: https://github.com/apache/airflow/pull/5048#discussion_r278708077
 
 

 ##
 File path: airflow/utils/log/es_task_handler.py
 ##
 @@ -67,19 +72,37 @@ def __init__(self, base_log_folder, filename_template,
 
 self.mark_end_on_close = True
 self.end_of_log_mark = end_of_log_mark
+self.write_stdout = write_stdout
+self.json_format = json_format
+self.json_fields = [label.strip() for label in json_fields.split(",")]
+
+self.handler = None
 
 def _render_log_id(self, ti, try_number):
 if self.log_id_jinja_template:
 jinja_context = ti.get_template_context()
 jinja_context['try_number'] = try_number
 return self.log_id_jinja_template.render(**jinja_context)
 
+if self.json_format:
+execution_date = self._clean_execution_date(ti.execution_date)
+else:
+execution_date = ti.execution_date.isoformat()
 return self.log_id_template.format(dag_id=ti.dag_id,
task_id=ti.task_id,
-   execution_date=ti
-   .execution_date.isoformat(),
+   execution_date=execution_date,
try_number=try_number)
 
+@staticmethod
+def _clean_execution_date(execution_date):
+"""
+Clean up an execution date so that it is safe to query in elasticsearch
+by removing reserved characters.
+# 
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters
+:param execution_date: execution date of the dag run.
+"""
+return re.sub(r"[\+\-\:\.]", "", execution_date.isoformat())
 
 Review comment:
   Why not use pendulum for datetime formatting? This is prone to errors


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5185: [AIRFLOW-4300] Task Modal Broken for DAGs that have not run

codecov-io edited a comment on issue #5185: [AIRFLOW-4300] Task Modal Broken 
for DAGs that have not run
URL: https://github.com/apache/airflow/pull/5185#issuecomment-486813713
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=h1) 
Report
   > Merging 
[#5185](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d333ffa5cc4f79f36117d4b33b546da9091e?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5185/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5185   +/-   ##
   ===
 Coverage   78.48%   78.48%   
   ===
 Files 467  467   
 Lines   2978229782   
   ===
 Hits2337323373   
 Misses   6409 6409
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5185/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=)
 | `82.27% <0%> (-1.27%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5185/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.59% <0%> (+0.17%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=footer). 
Last update 
[d333ffa...8409833](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5185: [AIRFLOW-4300] Task Modal Broken for DAGs that have not run

codecov-io commented on issue #5185: [AIRFLOW-4300] Task Modal Broken for DAGs 
that have not run
URL: https://github.com/apache/airflow/pull/5185#issuecomment-486813713
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=h1) 
Report
   > Merging 
[#5185](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d333ffa5cc4f79f36117d4b33b546da9091e?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5185/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5185   +/-   ##
   ===
 Coverage   78.48%   78.48%   
   ===
 Files 467  467   
 Lines   2978229782   
   ===
 Hits2337323373   
 Misses   6409 6409
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5185/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=)
 | `82.27% <0%> (-1.27%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5185/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.59% <0%> (+0.17%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=footer). 
Last update 
[d333ffa...8409833](https://codecov.io/gh/apache/airflow/pull/5185?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app.p`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 

  was:
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my best understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 [SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 

  was:
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my best understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or webserver_config.py that is boolean 
> for making Flask session permanent. Set session permanent in Flask 
> `before_request` as found online/stackoverflow. Setting session permanent on 
> login/signup is

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

{{```}}
 \{{ from flask import session as g_sesion}}
 \{{ ... }}
 {{ app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')}}
 {{ app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')}}{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
 \{{   @app.before_request}}
 \{{   def modify_request():}}
 \{{     logging.info("making session perm...")}}
 \{{     if session:}}
 \{{       session.permanent = True}}
 \{{    else:}}
 \{{       g_sesion.permanent = True}}{{if config:}}
 \{{ app.config.from_mapping(config)}}
 \{{ ...}}
 \{{  }}
 \{{ ```}}
 \{{  }}


*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 

  was:
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

{{```}}
{{ from flask import session as g_sesion}}
{{ ... }}
{{ app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')}}
{{ app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')}}{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
{{   @app.before_request}}
{{   def modify_request():}}
{{     logging.info("making session perm...")}}
{{     if session:}}
{{       session.permanent = True}}
{{    else:}}
{{       g_sesion.permanent = True}}{{if config:}}
{{ app.config.from_mapping(config)}}
{{ ...}}
{{  }}
{{ ```}}
{{  }}
{{ *Background*}}

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my bets understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or webserver_config.py that is boolean 
> for making Flask session permanent. Set session permanent in Flask 
> `before_request` as found online/stackoverflow. Setting session permanent on 
> login/signup is difficult given the different login options from FAB + 
> ability to plugin own auth into FAB.
>

[jira] [Created] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config

Joshua Kornblum created AIRFLOW-4413:


 Summary: Expose making flask sessions permanent via 
webserver_config
 Key: AIRFLOW-4413
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
 Project: Apache Airflow
  Issue Type: Improvement
  Components: webserver
Affects Versions: 1.10.3
Reporter: Joshua Kornblum
 Fix For: 1.10.4


(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

```
from flask import session as g_sesion
... 
app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 'COOKIE_SAMESITE')

if conf.getboolean('webserver','PERMANENT_SESSION'):
  @app.before_request
  def modify_request():
    logging.info("making session perm...")
    if session:
      session.permanent = True
   else:
      g_sesion.permanent = True

if config:
app.config.from_mapping(config)
...
 
```
 
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session). 

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app.py`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
I'm glad to complete and submit a PR after some feedback as this would be my 
first Airflow commit.

*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 

  was:
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app.py`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my best understanding of

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app.py`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 

  was:
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app.p`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my best understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

[SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 

  was:
(These comments reflect my best understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
  from flask import session as g_sesion
  ... 
  app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
  app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
    @app.before_request
    def modify_request():
      logging.info("making session perm...")
      if session:
       session.permanent = True
     else:
        g_sesion.permanent = True

{{if config:}}
   app.config.from_mapping(config)
 ...
  
 ```
{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 [SO - set Flask 
session|[https://stackoverflow.com/questions/34118093/flask-permanent-session-where-to-define-them]]

[Flask Config 
Docs|[http://flask.pocoo.org/docs/1.0/config/#PERMANENT_SESSION_LIFETIME]]

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my best understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`
{quote}{{```}}
 from flask import session as g_sesion
 ... 
 app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
 app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')

{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
   @app.before_request
   def modify_request():
     logging.info("making session perm...")
     if session:
      session.permanent = True
    else:
       g_sesion.permanent = True

{{if config:}}
  app.config.from_mapping(config)
...
 
```


{quote}
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 

  was:
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

{{```}}
 \{{ from flask import session as g_sesion}}
 \{{ ... }}
 {{ app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')}}
 {{ app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')}}{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
 \{{   @app.before_request}}
 \{{   def modify_request():}}
 \{{     logging.info("making session perm...")}}
 \{{     if session:}}
 \{{       session.permanent = True}}
 \{{    else:}}
 \{{       g_sesion.permanent = True}}{{if config:}}
 \{{ app.config.from_mapping(config)}}
 \{{ ...}}
 \{{  }}
 \{{ ```}}
 \{{  }}


*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my bets understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or webserver_config.py that is boolean 
> for making Flask session permanent. Set session permanent in Flask 
> `before_request` as found online/stackoverflow. Setting session permanent on 
> login/signup is difficult given the different login options from FAB + 
> ability to plugin own auth into FAB.
>  
> The change would look something like this ...
> In

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Description: 
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

{{```}}
{{ from flask import session as g_sesion}}
{{ ... }}
{{ app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')}}
{{ app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
'COOKIE_SAMESITE')}}{{if conf.getboolean('webserver','PERMANENT_SESSION'):}}
{{   @app.before_request}}
{{   def modify_request():}}
{{     logging.info("making session perm...")}}
{{     if session:}}
{{       session.permanent = True}}
{{    else:}}
{{       g_sesion.permanent = True}}{{if config:}}
{{ app.config.from_mapping(config)}}
{{ ...}}
{{  }}
{{ ```}}
{{  }}
{{ *Background*}}

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session).

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 

  was:
(These comments reflect my bets understanding of Airflow/Flask/FAB, if 
something is incorrect please let me know)

*Proposal*

Create setting in either airflow.cfg or webserver_config.py that is boolean for 
making Flask session permanent. Set session permanent in Flask `before_request` 
as found online/stackoverflow. Setting session permanent on login/signup is 
difficult given the different login options from FAB + ability to plugin own 
auth into FAB.

 

The change would look something like this ...

In `airflow.www.app`

```
from flask import session as g_sesion
... 
app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
'COOKIE_SECURE')
app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 'COOKIE_SAMESITE')

if conf.getboolean('webserver','PERMANENT_SESSION'):
  @app.before_request
  def modify_request():
    logging.info("making session perm...")
    if session:
      session.permanent = True
   else:
      g_sesion.permanent = True

if config:
app.config.from_mapping(config)
...
 
```
 
*Background*

With Flask default configuration the Flask session is non-permanent (destroyed 
when closing browser), the session lifetime is set to 31 days, and the session 
lifetime cannot be refreshed from activity (requires permanent session). 

At my employer we have corporate standard that application logins must expire 
after 12 hours of inactivity. I can set Flask config 
`PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
`session.permanent = True` the session cannot be refreshed from activity so the 
session lifetime I set starts ticking from session start regardless of 
activity. I feel like adding this security feature to config could be useful 
for many individuals managing Airflow deployments.

 

 

 


> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my bets understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or webserver_config.py that is boolean 
> for making Flask session permanent. Set session permanent in Flask 
> `before_request` as found online/stackoverflow. Setting session permanent on 
> login/signup is difficult given the different login options from FAB + 
> ability to plugin own auth into FAB.
>  
> The change would look something like this ...
> In `airflow.www.app`
> {{```}}
> {{ from flask import session as

[jira] [Updated] (AIRFLOW-4413) Expose making flask sessions permanent via webserver_config or airflow.cfg



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Kornblum updated AIRFLOW-4413:
-
Summary: Expose making flask sessions permanent via webserver_config or 
airflow.cfg  (was: Expose making flask sessions permanent via webserver_config)

> Expose making flask sessions permanent via webserver_config or airflow.cfg
> --
>
> Key: AIRFLOW-4413
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4413
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.3
>Reporter: Joshua Kornblum
>Priority: Minor
> Fix For: 1.10.4
>
>
> (These comments reflect my bets understanding of Airflow/Flask/FAB, if 
> something is incorrect please let me know)
> *Proposal*
> Create setting in either airflow.cfg or webserver_config.py that is boolean 
> for making Flask session permanent. Set session permanent in Flask 
> `before_request` as found online/stackoverflow. Setting session permanent on 
> login/signup is difficult given the different login options from FAB + 
> ability to plugin own auth into FAB.
>  
> The change would look something like this ...
> In `airflow.www.app`
> ```
> from flask import session as g_sesion
> ... 
> app.config['SESSION_COOKIE_SECURE'] = conf.getboolean('webserver', 
> 'COOKIE_SECURE')
> app.config['SESSION_COOKIE_SAMESITE'] = conf.get('webserver', 
> 'COOKIE_SAMESITE')
> if conf.getboolean('webserver','PERMANENT_SESSION'):
>   @app.before_request
>   def modify_request():
>     logging.info("making session perm...")
>     if session:
>       session.permanent = True
>    else:
>       g_sesion.permanent = True
> if config:
> app.config.from_mapping(config)
> ...
>  
> ```
>  
> *Background*
> With Flask default configuration the Flask session is non-permanent 
> (destroyed when closing browser), the session lifetime is set to 31 days, and 
> the session lifetime cannot be refreshed from activity (requires permanent 
> session). 
> At my employer we have corporate standard that application logins must expire 
> after 12 hours of inactivity. I can set Flask config 
> `PERMANENT_SESSION_LIFETIME` and Flask will respect that but without setting 
> `session.permanent = True` the session cannot be refreshed from activity so 
> the session lifetime I set starts ticking from session start regardless of 
> activity. I feel like adding this security feature to config could be useful 
> for many individuals managing Airflow deployments.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] schnie commented on a change in pull request #5048: [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler

schnie commented on a change in pull request #5048: [AIRFLOW-3370] Add stdout 
output options to Elasticsearch task log handler
URL: https://github.com/apache/airflow/pull/5048#discussion_r278682368
 
 

 ##
 File path: airflow/config_templates/default_airflow.cfg
 ##
 @@ -592,6 +592,9 @@ hide_sensitive_variable_fields = True
 elasticsearch_host =
 elasticsearch_log_id_template = 
{{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
 elasticsearch_end_of_log_mark = end_of_log
+elasticsearch_write_stdout = False
+elasticsearch_json_format = False
+elasticsearch_json_fields = asctime, filename, lineno, levelname, message
 
 Review comment:
   @KevinYang21 good call! Added some comments, hope they help!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] r39132 merged pull request #5186: [AIRFLOW-XXX] Add Bill DeRose to Jetlore in ReadMe

r39132 merged pull request #5186: [AIRFLOW-XXX] Add Bill DeRose to Jetlore in 
ReadMe
URL: https://github.com/apache/airflow/pull/5186
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] r39132 opened a new pull request #5186: [AIRFLOW-XXX] Add Bill DeRose to Jetlore in ReadMe

r39132 opened a new pull request #5186: [AIRFLOW-XXX] Add Bill DeRose to 
Jetlore in ReadMe
URL: https://github.com/apache/airflow/pull/5186
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4300) Task Modal Broken for DAGs that have not run



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826296#comment-16826296
 ] 

ASF GitHub Bot commented on AIRFLOW-4300:
-

marcusianlevine commented on pull request #5185: [AIRFLOW-4300] Task Modal 
Broken for DAGs that have not run
URL: https://github.com/apache/airflow/pull/5185
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4300
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Fixes a bug that prevents the task detail modal from appearing in the Graph 
view on DAGs that have not yet run
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Pure JavaScript change to fix a known bug
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Task Modal Broken for DAGs that have not run
> 
>
> Key: AIRFLOW-4300
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4300
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Major
> Fix For: 1.10.4
>
>
> After upgrading to 1.10.3, the task detail modal that normally appears when 
> clicking on a task from the graph view does not appear when the DAG has not 
> been run yet
> This makes it impossible to see the task details and rendered template prior 
> to running the DAG
> In the JavaScript console, it appears this issue is caused by 'try_number' 
> being undefined when the DAG has not been run yet:
> {code:java}
> Uncaught TypeError: Cannot read property 'try_number' of undefined
> at SVGGElement. (graph?dag_id=my-dag:1)
> at SVGGElement.__onclick (d3.v3.min.js:1)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] marcusianlevine opened a new pull request #5185: [AIRFLOW-4300] Task Modal Broken for DAGs that have not run

marcusianlevine opened a new pull request #5185: [AIRFLOW-4300] Task Modal 
Broken for DAGs that have not run
URL: https://github.com/apache/airflow/pull/5185
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4300
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Fixes a bug that prevents the task detail modal from appearing in the Graph 
view on DAGs that have not yet run
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Pure JavaScript change to fix a known bug
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-4306) Global operator extra links

2019-04-25 Thread Tao Feng (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng resolved AIRFLOW-4306.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> Global operator extra links
> ---
>
> Key: AIRFLOW-4306
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4306
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> A way to register global operator extra links that are shared by all the 
> operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kunalyogenshah commented on issue #5164: [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI.

2019-04-25 Thread ASF subversion and git services (JIRA)

kunalyogenshah commented on issue #5164: [AIRFLOW-4396] Provide a link to 
external Elasticsearch logs in UI.
URL: https://github.com/apache/airflow/pull/5164#issuecomment-486761108
 
 
   Bringing this thread back up to share some alternate thoughts. I'd like to 
advocate for this approach specifically for just logs look up for the following 
reasons : 
- Using elasticsearch for now is quite well baked into the log handling, 
and Kibana is possibly the most common frontend for this. Using that as a 
frontend for visualizing / searching logs seems like quite a useful central 
feature to have as compared to a plugin. Semantically we'd use extra links 
plugin for features that are more specific to each individual Airflow DAG / 
Task / Provider.
- This can be quite easily integrated and extended like the existing 
download logs / view logs, and seems on the same plane w.r.t. utility.
- Also referring to the proposed metrics changes in this PR 
(https://github.com/apache/airflow/pull/5139), having a standalone endpoint for 
serving logs allows us to track metrics for just that endpoint better than a 
common extra links section. (like tracking views and failures, etc). 
   
   Thoughts?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4306) Global operator extra links



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826236#comment-16826236
 ] 

ASF subversion and git services commented on AIRFLOW-4306:
--

Commit d333ffa5cc4f79f36117d4b33b546da9091e in airflow's branch 
refs/heads/master from Chao-Han Tsai
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d333ffa ]

[AIRFLOW-4306] Global operator extra links (#5094)

Provide a way to register global operator extra
links through airflow plugins.

> Global operator extra links
> ---
>
> Key: AIRFLOW-4306
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4306
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> A way to register global operator extra links that are shared by all the 
> operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4306) Global operator extra links



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826235#comment-16826235
 ] 

ASF GitHub Bot commented on AIRFLOW-4306:
-

feng-tao commented on pull request #5094: [AIRFLOW-4306] Global operator extra 
links
URL: https://github.com/apache/airflow/pull/5094
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Global operator extra links
> ---
>
> Key: AIRFLOW-4306
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4306
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> A way to register global operator extra links that are shared by all the 
> operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao merged pull request #5094: [AIRFLOW-4306] Global operator extra links

feng-tao merged pull request #5094: [AIRFLOW-4306] Global operator extra links
URL: https://github.com/apache/airflow/pull/5094
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5177: [AIRFLOW-4084] Fix bug downloading incomplete logs from ElasticSearch

codecov-io commented on issue #5177: [AIRFLOW-4084] Fix bug downloading 
incomplete logs from ElasticSearch
URL: https://github.com/apache/airflow/pull/5177#issuecomment-486749719
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@3becc60`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5177/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#5177   +/-   ##
   =
 Coverage  ?   10.73%   
   =
 Files ?  467   
 Lines ?29789   
 Branches  ?0   
   =
 Hits  ? 3198   
 Misses?26591   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/5177/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (ø)` | |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5177/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `0% <0%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=footer). 
Last update 
[3becc60...288437f](https://codecov.io/gh/apache/airflow/pull/5177?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] languitar commented on issue #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or webhook_token, or both

languitar commented on issue #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can 
take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066#issuecomment-486747994
 
 
   Any chance to get this into a new stable release? The SlackWebhookHook ist 
pretty unusable to me without this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-4300) Task Modal Broken for DAGs that have not run

2019-04-25 Thread Marcus Levine (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-4300 started by Marcus Levine.
--
> Task Modal Broken for DAGs that have not run
> 
>
> Key: AIRFLOW-4300
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4300
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Major
> Fix For: 1.10.4
>
>
> After upgrading to 1.10.3, the task detail modal that normally appears when 
> clicking on a task from the graph view does not appear when the DAG has not 
> been run yet
> This makes it impossible to see the task details and rendered template prior 
> to running the DAG
> In the JavaScript console, it appears this issue is caused by 'try_number' 
> being undefined when the DAG has not been run yet:
> {code:java}
> Uncaught TypeError: Cannot read property 'try_number' of undefined
> at SVGGElement. (graph?dag_id=my-dag:1)
> at SVGGElement.__onclick (d3.v3.min.js:1)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-4300) Task Modal Broken for DAGs that have not run

2019-04-25 Thread Marcus Levine (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Levine reassigned AIRFLOW-4300:
--

Assignee: Marcus Levine

> Task Modal Broken for DAGs that have not run
> 
>
> Key: AIRFLOW-4300
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4300
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Major
> Fix For: 1.10.4
>
>
> After upgrading to 1.10.3, the task detail modal that normally appears when 
> clicking on a task from the graph view does not appear when the DAG has not 
> been run yet
> This makes it impossible to see the task details and rendered template prior 
> to running the DAG
> In the JavaScript console, it appears this issue is caused by 'try_number' 
> being undefined when the DAG has not been run yet:
> {code:java}
> Uncaught TypeError: Cannot read property 'try_number' of undefined
> at SVGGElement. (graph?dag_id=my-dag:1)
> at SVGGElement.__onclick (d3.v3.min.js:1)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-3653) Pausing a Dag does not pause its SubDags



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Gavrilenko reassigned AIRFLOW-3653:


Assignee: Vladimir Gavrilenko

> Pausing a Dag does not pause its SubDags
> 
>
> Key: AIRFLOW-3653
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3653
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Håvard Wall
>Assignee: Vladimir Gavrilenko
>Priority: Major
>
> Pausing a Dag will have no effect on running SubDags and let them complete 
> fully.
> Expected behavior: A running subdag will complete it's current running task, 
> but not start a new one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] nikolayskovpin commented on a change in pull request #5152: [AIRFLOW-4385] Add namespace into pod's yaml

nikolayskovpin commented on a change in pull request #5152: [AIRFLOW-4385] Add 
namespace into pod's yaml
URL: https://github.com/apache/airflow/pull/5152#discussion_r278610804
 
 

 ##
 File path: 
airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py
 ##
 @@ -75,6 +77,7 @@ class ExtractXcomPodRequestFactory(KubernetesRequestFactory):
 _yaml = """apiVersion: v1
 kind: Pod
 metadata:
+  namespace: default
 
 Review comment:
   default


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-4403) UI search query in main view should filters by either dag_id or owner



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4403:
---
Summary: UI search query in main view should filters by either dag_id or 
owner  (was: UI search query in main view filters only dag_ids)

> UI search query in main view should filters by either dag_id or owner
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
> Fix For: 1.10.4
>
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-4403) UI search query in main view filters only dag_ids



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG resolved AIRFLOW-4403.

   Resolution: Resolved
Fix Version/s: 1.10.4

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
> Fix For: 1.10.4
>
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids

2019-04-25 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826139#comment-16826139
 ] 

ASF GitHub Bot commented on AIRFLOW-4403:
-

XD-DENG commented on pull request #5184: [AIRFLOW-4403] search by `dag_id` or 
`owners`
URL: https://github.com/apache/airflow/pull/5184
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826141#comment-16826141
 ] 

ASF subversion and git services commented on AIRFLOW-4403:
--

Commit 8f7ea43c473e17473b0590090b541bbfa5747bee in airflow's branch 
refs/heads/master from Vladimir Gavrilenko
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8f7ea43 ]

[AIRFLOW-4403] search by `dag_id` or `owners` in UI (#5184)

- filtering on both columns restored, not only on `dag_id`.

This was the default bahavior, but broken in 
https://github.com/apache/airflow/commit/a62bc75fce1bb9a1700c55e8cbd24c7c9400fa80.

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] XD-DENG merged pull request #5184: [AIRFLOW-4403] search by `dag_id` or `owners`

XD-DENG merged pull request #5184: [AIRFLOW-4403] search by `dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`

XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184#issuecomment-486706360
 
 
   Oh no @vladimir-gavrilenko . please ignore my msgs above. There is nothing 
wrong with your change in terms of styling, and nothing broken. I just made 
some change for comparing locally then forgot to change back.
   
   My bad!
   
   Will merge now. Thanks for taking this up!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4412) PodLauncher drops log lines printed by the pod

2019-04-25 Thread Ondrej Holecek (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826118#comment-16826118
 ] 

Ondrej Holecek commented on AIRFLOW-4412:
-

I propose the fix:[

https://github.com/smrt28/airflow/commit/abd8912d68ecc8c17abb606f09dba2dcc9d55118]

> PodLauncher drops log lines printed by the pod
> --
>
> Key: AIRFLOW-4412
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4412
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: kubernetes
>Affects Versions: 1.10.3
>Reporter: Ondrej Holecek
>Priority: Major
>
> When PodLauncher starts a pod, it is checking the pod status by
> pooling read_pod() method in 1s intervals until the pod state != QUEUE.
>  After the pod starts, it tails the pod logs until the pod finishes. Due
>  to the 1s pooling interval, there could be up to 1s delay between the
>  real pod startup and the log tailing, and due to "tail_lines=10" all the
>  lines above those last 10 logged within the 1s interval would be lost.
>  
> The lines read from the stream returned by read_namespaced_pod_log()
>  should be converted to string and right-stripped, since they contain
>  '\n' character at the end.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`

XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184#issuecomment-486703593
 
 
   I observe Flake8 error below for your change:
   ```
   ./airflow/www/views.py:241:20: E131 continuation line unaligned for hanging 
indent
   ```
   
   BTW, I have tested your change locally, the functionality is correct  Now 
we just need to address style issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`

XD-DENG commented on issue #5184: [AIRFLOW-4403] search by `dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184#issuecomment-486701278
 
 
   @vladimir-gavrilenko  
   The change is correct in terms of logic. But it can not pass Flake8 style 
check (the Flake8 check in the Travis CI seems broken, I will check).
   
   Please change accordingly to ensure your change can pass flake8 check 
locally first please. Later I may need you to rebase and re-commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (AIRFLOW-4412) PodLauncher drops log lines printed by the pod

2019-04-25 Thread Ondrej Holecek (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826118#comment-16826118
 ] 

Ondrej Holecek edited comment on AIRFLOW-4412 at 4/25/19 2:30 PM:
--

I propose the fix:

[https://github.com/smrt28/airflow/commit/abd8912d68ecc8c17abb606f09dba2dcc9d55118]]


was (Author: smrt28):
I propose the fix:[

https://github.com/smrt28/airflow/commit/abd8912d68ecc8c17abb606f09dba2dcc9d55118]

> PodLauncher drops log lines printed by the pod
> --
>
> Key: AIRFLOW-4412
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4412
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: kubernetes
>Affects Versions: 1.10.3
>Reporter: Ondrej Holecek
>Priority: Major
>
> When PodLauncher starts a pod, it is checking the pod status by
> pooling read_pod() method in 1s intervals until the pod state != QUEUE.
>  After the pod starts, it tails the pod logs until the pod finishes. Due
>  to the 1s pooling interval, there could be up to 1s delay between the
>  real pod startup and the log tailing, and due to "tail_lines=10" all the
>  lines above those last 10 logged within the 1s interval would be lost.
>  
> The lines read from the stream returned by read_namespaced_pod_log()
>  should be converted to string and right-stripped, since they contain
>  '\n' character at the end.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-4412) PodLauncher drops log lines printed by the pod

2019-04-25 Thread Ondrej Holecek (JIRA)

Ondrej Holecek created AIRFLOW-4412:
---

 Summary: PodLauncher drops log lines printed by the pod
 Key: AIRFLOW-4412
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4412
 Project: Apache Airflow
  Issue Type: Bug
  Components: kubernetes
Affects Versions: 1.10.3
Reporter: Ondrej Holecek


When PodLauncher starts a pod, it is checking the pod status by
pooling read_pod() method in 1s intervals until the pod state != QUEUE.
 After the pod starts, it tails the pod logs until the pod finishes. Due
 to the 1s pooling interval, there could be up to 1s delay between the
 real pod startup and the log tailing, and due to "tail_lines=10" all the
 lines above those last 10 logged within the 1s interval would be lost.
 
The lines read from the stream returned by read_namespaced_pod_log()
 should be converted to string and right-stripped, since they contain
 '\n' character at the end.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids

2019-04-25 Thread Damien Avrillon (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826088#comment-16826088
 ] 

Damien Avrillon commented on AIRFLOW-4403:
--

[~Vladimir Gavrilenko], [~XD-DENG],

Thank you for your responsiveness.

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4411) Configuration keys are set to lowercase

2019-04-25 Thread Viktor Bogdanov (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Bogdanov updated AIRFLOW-4411:
-
Description: 
We are using kubernetes node selectors with keys `InstanceGroup = general` But 
airflow sets all keys to lowercase 
[https://github.com/apache/airflow/blob/603d02a48c9fa40bd303f458b6a1cf1363aebab3/airflow/configuration.py#L207]
 Because of this in result pod spec we have wrong selectors `Node-Selectors:  
instancegroup=general`

  was:
We are using kubernetes node selectors with keys `InstanceGroup = general` But 
airflow sets all keys to lowecase 
https://github.com/apache/airflow/blob/603d02a48c9fa40bd303f458b6a1cf1363aebab3/airflow/configuration.py#L207
Because of this in result pod spec we have wrong selectors `Node-Selectors:  
instancegroup=general`


> Configuration keys are set to lowercase
> ---
>
> Key: AIRFLOW-4411
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4411
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.10.3
>Reporter: Viktor Bogdanov
>Priority: Minor
>
> We are using kubernetes node selectors with keys `InstanceGroup = general` 
> But airflow sets all keys to lowercase 
> [https://github.com/apache/airflow/blob/603d02a48c9fa40bd303f458b6a1cf1363aebab3/airflow/configuration.py#L207]
>  Because of this in result pod spec we have wrong selectors `Node-Selectors:  
> instancegroup=general`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826001#comment-16826001
 ] 

ASF GitHub Bot commented on AIRFLOW-4403:
-

vladimir-gavrilenko commented on pull request #5184: [AIRFLOW-4403] search by 
`dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Filtering on both columns restored, not only on `dag_id`
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
   @XD-DENG, could you please have a look at it?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] vladimir-gavrilenko opened a new pull request #5184: [AIRFLOW-4403] search by `dag_id` or `owners`

vladimir-gavrilenko opened a new pull request #5184: [AIRFLOW-4403] search by 
`dag_id` or `owners`
URL: https://github.com/apache/airflow/pull/5184
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Filtering on both columns restored, not only on `dag_id`
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
   @XD-DENG, could you please have a look at it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] raviagarwalunravel commented on issue #5118: [AIRFLOW-4315] Add monitoring API's to airflow

raviagarwalunravel commented on issue #5118: [AIRFLOW-4315] Add monitoring 
API's to airflow
URL: https://github.com/apache/airflow/pull/5118#issuecomment-486636829
 
 
   If anybody knows about the build issues this PR is facing, please help me 
out.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-4403) UI search query in main view filters only dag_ids



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Gavrilenko reassigned AIRFLOW-4403:


Assignee: Vladimir Gavrilenko

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825934#comment-16825934
 ] 

Xiaodong DENG commented on AIRFLOW-4403:


[~Vladimir Gavrilenko]

I have done a quick check: yes, the search feature should filter by DAG ID or 
OWNER. But this feature was broken in 
[https://github.com/apache/airflow/commit/a62bc75fce1bb9a1700c55e8cbd24c7c9400fa80]
 

If you would like to take it, please assign it to yourself here. For the PR, 
please raise your PR against `master` branch only. You can tag me there (ID: 
XD-DENG) when it's ready for review.

Thanks! 

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825936#comment-16825936
 ] 

Vladimir Gavrilenko commented on AIRFLOW-4403:
--

[~XD-DENG]

Ok, thank you!

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Assignee: Vladimir Gavrilenko
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825932#comment-16825932
 ] 

Vladimir Gavrilenko commented on AIRFLOW-4403:
--

Yep, if you confirm that filtering should include oner id also, I'd like to 
take it.

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825927#comment-16825927
 ] 

Xiaodong DENG commented on AIRFLOW-4403:


[~Vladimir Gavrilenko] I'm not sure.

I tested in my 1.10.0 env, and it does search by Owner as well.

I can double-confirm then get back here. You're interested to take this up?

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] codecov-io commented on issue #5079: [WIP][AIRFLOW-4285] Update task dependency context defination and usage

codecov-io commented on issue #5079: [WIP][AIRFLOW-4285] Update task dependency 
context defination and usage
URL: https://github.com/apache/airflow/pull/5079#issuecomment-486600978
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=h1) 
Report
   > Merging 
[#5079](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/603d02a48c9fa40bd303f458b6a1cf1363aebab3?src=pr=desc)
 will **increase** coverage by `0.04%`.
   > The diff coverage is `97.1%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5079/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #5079  +/-   ##
   =
   + Coverage   78.46%   78.5%   +0.04% 
   =
 Files 467 469   +2 
 Lines   29766   29804  +38 
   =
   + Hits23357   23399  +42 
   + Misses   64096405   -4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `96.15% <100%> (+0.15%)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.33% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.88% <100%> (+0.45%)` | :arrow_up: |
   | 
[airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `66.63% <50%> (ø)` | :arrow_up: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `78.68% <94.73%> (+0.15%)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `96% <96%> (ø)` | |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `59.05% <0%> (-0.18%)` | :arrow_down: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/5079/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=footer). 
Last update 
[603d02a...f9549fe](https://codecov.io/gh/apache/airflow/pull/5079?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2439) Official Docker image

2019-04-25 Thread Johannes Wienke (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825900#comment-16825900
 ] 

Johannes Wienke commented on AIRFLOW-2439:
--

It is also currently pretty confusing to find an apache/airflow image without 
any stable tags to depend on.

> Official Docker image
> -
>
> Key: AIRFLOW-2439
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2439
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>
> I think we need an official Airflow Docker image, especially with the 
> creation of the KubernetesExecutor. There's lots of 3rd party Airflow Docker 
> images, but now that we have example Kubernetes resources which require an 
> image to be specified, it would benefit to have an official Docker image 
> (instead of first having to build your own).
> To request an official Apache Airflow Docker Hub account, we need to file an 
> official repository PR at the docker-library repo. I would like to set up and 
> file the request, and there might be some back-and-forth discussion before 
> the proposal is accepted. [More 
> info|https://docs.docker.com/docker-hub/official_repos/#how-do-i-create-a-new-official-repository]
> I like the setup Flink has done by creating a separate docker-flink account 
> in which the docker-library repos are forked, and a repo with scripts e.g. 
> for generating the docker library manifest: [https://github.com/docker-flink].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (AIRFLOW-4403) UI search query in main view filters only dag_ids



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Gavrilenko updated AIRFLOW-4403:
-
Comment: was deleted

(was: As I can see in the code, searching is done only by dag_id:

[https://github.com/apache/airflow/blob/master/airflow/www/views.py#L238-L242]

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/www_rbac/views.py#L235-L236]

whereas at v1.10.2 searching included both dag_id and owner:

[https://github.com/apache/airflow/blob/1.10.2/airflow/www/views.py#L2173-L2181])

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4403) UI search query in main view filters only dag_ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825884#comment-16825884
 ] 

Vladimir Gavrilenko commented on AIRFLOW-4403:
--

As I can see in the code, searching is done only by dag_id:

[https://github.com/apache/airflow/blob/master/airflow/www/views.py#L238-L242]

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/www_rbac/views.py#L235-L236]

whereas at v1.10.2 searching included both dag_id and owner:

[https://github.com/apache/airflow/blob/1.10.2/airflow/www/views.py#L2173-L2181]

> UI search query in main view filters only dag_ids
> -
>
> Key: AIRFLOW-4403
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4403
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.3
>Reporter: Damien Avrillon
>Priority: Minor
>
> In airflow 1.10.2, In main dags view, UI search was filtering not only 'DAG' 
> but also 'Owner' column.
> For example, if there are multiple dags / owners:
> dag_id_1    /   owner_id_1
> dag_id_2    /   owner_id_2
> dag_id_3    /   owner_id_2
> Searching for '_2' would return 2 dags: 'dag_id_2' and 'dag_id_3' because 
> 'Owner' matched.
> As upgrading to 1.10.3, the same search yields one single result: 'dag_id_2'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] XD-DENG commented on issue #5172: [AIRFLOW-4324] fix search in RBAC UI - master

XD-DENG commented on issue #5172: [AIRFLOW-4324] fix search in RBAC UI - master
URL: https://github.com/apache/airflow/pull/5172#issuecomment-486582874
 
 
   Hi @vladimir-gavrilenko ,
   
   - in `v1-10-stable` we still two sets of UI, `/www` and `/www_rbac`.
   
   - in master, we only have one UI, `/www`, but actually it's the previous 
`/www_rbac`.
   
   So this fix should have already been applied to all places ;-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] vladimir-gavrilenko edited a comment on issue #5172: [AIRFLOW-4324] fix search in RBAC UI - master

vladimir-gavrilenko edited a comment on issue #5172: [AIRFLOW-4324] fix search 
in RBAC UI - master
URL: https://github.com/apache/airflow/pull/5172#issuecomment-48657
 
 
   @XD-DENG 
   Shouldn't it be merged to `v1-10-stable` also?
   As I can see, the original PR fixed the problem in RBAC UI on `v1-10-stable` 
branch and this PR fixes the main UI on `master`
   
https://github.com/apache/airflow/blob/v1-10-stable/airflow/www/views.py#L2149


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] vladimir-gavrilenko commented on issue #5172: [AIRFLOW-4324] fix search in RBAC UI - master

vladimir-gavrilenko commented on issue #5172: [AIRFLOW-4324] fix search in RBAC 
UI - master
URL: https://github.com/apache/airflow/pull/5172#issuecomment-48657
 
 
   @XD-DENG 
   Shouldn't it be merged to `v1-10-stable` also?
   As I can see, the original PR fixed the problem in RBAC UI on `v1-10-stable` 
branch and this PR fixes the main UI on `master`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4410) Ldap authentication failed when using non-ssl ldap server



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825871#comment-16825871
 ] 

ASF GitHub Bot commented on AIRFLOW-4410:
-

Jerevia commented on pull request #5183: [AIRFLOW-4410]Add Non-ssl ldap server 
support
URL: https://github.com/apache/airflow/pull/5183
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Ldap authentication failed when using non-ssl ldap server
> -
>
> Key: AIRFLOW-4410
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4410
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: authentication
>Affects Versions: 1.10.1, 1.10.2, 1.10.3
> Environment: Python 3.6, apache-airflow==1.10.3
>Reporter: Jeremy
>Priority: Major
>
> I modified the aiflow.cfg in the $AIRFLOW_HOME with
>  
> {code:java}
> [webserver]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.ldap_auth
> [ldap]
> uri = 
> user_filter = objectclass=posixAccount
> user_name_attr = uid
> group_member_attr = ou
> superuser_filter =
> data_profiler_filter =
> bind_user =
> bind_password =
> basedn = 
> search_scope = SUBTREE{code}
>  
> And I started the airflow web server with command: 
> {code:java}
> airflow webserver{code}
> But when I signed in airflow in the login form, the following exception was 
> rasied:
>  
> {code:java}
> Traceback (most recent call last):
>  File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
> line 2292, in wsgi_app
>  response = self.full_dispatch_request()
>  File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
> line 1815, in full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
> line 1718, in handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File 
> "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/_compat.py", 
> line 35, in reraise
>  raise value
>  File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
> line 1813, in full_dispatch_request
>  rv = self.dispatch_request()
>  File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
> line 1799, in dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
>

[GitHub] [airflow] Jerevia opened a new pull request #5183: [AIRFLOW-4410]Add Non-ssl ldap server support

Jerevia opened a new pull request #5183: [AIRFLOW-4410]Add Non-ssl ldap server 
support
URL: https://github.com/apache/airflow/pull/5183
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links

codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator 
extra links
URL: https://github.com/apache/airflow/pull/5094#issuecomment-483039818
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=h1) 
Report
   > Merging 
[#5094](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/603d02a48c9fa40bd303f458b6a1cf1363aebab3?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `95.23%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5094/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5094  +/-   ##
   ==
   + Coverage   78.46%   78.47%   +<.01% 
   ==
 Files 467  467  
 Lines   2976629782  +16 
   ==
   + Hits2335723372  +15 
   - Misses   6409 6410   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <ø> (ø)` | :arrow_up: |
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `86.91% <100%> (+0.5%)` | :arrow_up: |
   | 
[airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=)
 | `87.03% <100%> (+0.24%)` | :arrow_up: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `93.93% <92.85%> (-0.1%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=footer). 
Last update 
[603d02a...72793a9](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links

codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator 
extra links
URL: https://github.com/apache/airflow/pull/5094#issuecomment-483039818
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=h1) 
Report
   > Merging 
[#5094](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/603d02a48c9fa40bd303f458b6a1cf1363aebab3?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `95.23%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5094/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5094  +/-   ##
   ==
   + Coverage   78.46%   78.47%   +<.01% 
   ==
 Files 467  467  
 Lines   2976629782  +16 
   ==
   + Hits2335723372  +15 
   - Misses   6409 6410   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <ø> (ø)` | :arrow_up: |
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `86.91% <100%> (+0.5%)` | :arrow_up: |
   | 
[airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=)
 | `87.03% <100%> (+0.24%)` | :arrow_up: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `93.93% <92.85%> (-0.1%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=footer). 
Last update 
[603d02a...72793a9](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4410) Ldap authentication failed when using non-ssl ldap server

2019-04-25 Thread Jeremy (JIRA)

Jeremy created AIRFLOW-4410:
---

 Summary: Ldap authentication failed when using non-ssl ldap server
 Key: AIRFLOW-4410
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4410
 Project: Apache Airflow
  Issue Type: Bug
  Components: authentication
Affects Versions: 1.10.3, 1.10.2, 1.10.1
 Environment: Python 3.6, apache-airflow==1.10.3
Reporter: Jeremy


I modified the aiflow.cfg in the $AIRFLOW_HOME with

 
{code:java}
[webserver]
authenticate = True
auth_backend = airflow.contrib.auth.backends.ldap_auth

[ldap]
uri = 
user_filter = objectclass=posixAccount
user_name_attr = uid
group_member_attr = ou
superuser_filter =
data_profiler_filter =
bind_user =
bind_password =
basedn = 
search_scope = SUBTREE{code}
 

And I started the airflow web server with command: 
{code:java}
airflow webserver{code}
But when I signed in airflow in the login form, the following exception was 
rasied:

 
{code:java}
Traceback (most recent call last):
 File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
line 2292, in wsgi_app
 response = self.full_dispatch_request()
 File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
line 1815, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
line 1718, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/_compat.py", 
line 35, in reraise
 raise value
 File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
line 1813, in full_dispatch_request
 rv = self.dispatch_request()
 File "/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask/app.py", 
line 1799, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask_admin/base.py", 
line 69, in inner
 return self._run_view(f, *args, **kwargs)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/flask_admin/base.py", 
line 368, in _run_view
 return fn(self, *args, **kwargs)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/airflow/www/views.py", 
line 731, in login
 return airflow.login.login(self, request)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/airflow/utils/db.py", 
line 73, in wrapper
 return func(*args, **kwargs)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
 line 308, in login
 LdapUser.try_login(username, password)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
 line 196, in try_login
 configuration.conf.get("ldap", "bind_password"))
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
 line 72, in get_ldap_connection
 ca_certs_file=cacert)
 File 
"/data/home/jeremy/anaconda3/lib/python3.6/site-packages/ldap3/core/tls.py", 
line 93, in __init__
 raise LDAPSSLConfigurationError('invalid CA public key file')
ldap3.core.exceptions.LDAPSSLConfigurationError: invalid CA public key file 
{code}
I think using non-ssl ldap server should be considered for the internal airflow 
server.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] XD-DENG merged pull request #5179: [AIRFLOW-XXX] Fix doc sphinx error

XD-DENG merged pull request #5179: [AIRFLOW-XXX] Fix doc sphinx error
URL: https://github.com/apache/airflow/pull/5179
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5179: [AIRFLOW-XXX] Fix doc sphinx error

codecov-io edited a comment on issue #5179: [AIRFLOW-XXX] Fix doc sphinx error
URL: https://github.com/apache/airflow/pull/5179#issuecomment-486528600
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@3becc60`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5179/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#5179   +/-   ##
   =
 Coverage  ?   78.47%   
   =
 Files ?  467   
 Lines ?29766   
 Branches  ?0   
   =
 Hits  ?23358   
 Misses? 6408   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5179/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <ø> (ø)` | |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5179/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <ø> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=footer). 
Last update 
[3becc60...36990a7](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5179: [AIRFLOW-XXX] Fix doc sphinx error

codecov-io edited a comment on issue #5179: [AIRFLOW-XXX] Fix doc sphinx error
URL: https://github.com/apache/airflow/pull/5179#issuecomment-486528600
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@3becc60`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5179/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#5179   +/-   ##
   =
 Coverage  ?   78.47%   
   =
 Files ?  467   
 Lines ?29766   
 Branches  ?0   
   =
 Hits  ?23358   
 Misses? 6408   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5179/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `74.93% <ø> (ø)` | |
   | 
[airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5179/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5)
 | `90.51% <ø> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=footer). 
Last update 
[3becc60...36990a7](https://codecov.io/gh/apache/airflow/pull/5179?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4361) Fix flaky test_integration_run_dag_with_scheduler_failure



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825816#comment-16825816
 ] 

ASF GitHub Bot commented on AIRFLOW-4361:
-

milton0825 commented on pull request #5182:  [AIRFLOW-4361] Fix flaky 
test_integration_run_dag_with_scheduler_failure
URL: https://github.com/apache/airflow/pull/5182
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix flaky test_integration_run_dag_with_scheduler_failure
> -
>
> Key: AIRFLOW-4361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4361
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> test_integration_run_dag_with_scheduler_failure often fails with
> {code}
> ^[[33m   ^[[33m^[[1m^[[33mConnectionError^[[0m^[[0m^[[33m: 
> ^[[0m^[[33mHTTPConnectionPool(host='10.20.3.19', port=  30809): Max 
> retries exceeded with url: 
> /api/experimental/dags/example_kubernetes_executor_config/paused/false (Ca
>   used by NewConnectionError(' 0x7fb80986f208>: Failed to establish a n  ew connection: [Errno 111] 
> Connection refused',))^[[0m^M
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] milton0825 opened a new pull request #5182: [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure