date:20200203

feluelle commented on issue #7347: [AIRFLOW-] Add pre-commit check for 
utf-8 file encoding
URL: https://github.com/apache/airflow/pull/7347#issuecomment-581786035
 
 
   True. 臘‍♂ 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added 
GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-581781667
 
 
   Thank you @nuclearpinguin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow

2020-02-03 Thread Chandu Kavar (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029616#comment-17029616
 ] 

Chandu Kavar commented on AIRFLOW-2906:
---

Yes, I have made some changes to my local and stuck in unit tests for that.

> DataDog Integration for Airflow
> ---
>
> Key: AIRFLOW-2906
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2906
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.8.0
>Reporter: Austin Hsu
>Assignee: Chandu Kavar
>Priority: Minor
>  Labels: metrics
>
> Add functionality to Airflow to enable sending of metrics to DataDog.  
> DataDog provides support for tags which allows us to aggregate data more 
> easily and visualize it.  We can utilize the [Datadog python 
> library|https://github.com/DataDog/datadogpy] python library and the [Datadog 
> ThreadStats 
> module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module]
>  to send metrics directly to DataDog without needing to spin up an agent to 
> forward the metrics.  The current implementation in 1.8 uses the statsd 
> library to send the metrics which provides us with much less control to 
> filter our data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process

codecov-io commented on issue #7232: [AIRFLOW-6569] Flush pending Sentry 
exceptions before exiting forked process
URL: https://github.com/apache/airflow/pull/7232#issuecomment-581760360
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=h1) 
Report
   > Merging 
[#7232](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c1ede4d63f6498d307ca5d0572296fc99d2eb6a2?src=pr=desc)
 will **increase** coverage by `0.67%`.
   > The diff coverage is `33.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7232/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7232  +/-   ##
   ==
   + Coverage   85.28%   85.96%   +0.67% 
   ==
 Files 761  871 +110 
 Lines   4000440634 +630 
   ==
   + Hits3411634929 +813 
   + Misses   5888 5705 -183
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `64.06% <0%> (-3.16%)` | :arrow_down: |
   | 
[airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9zZW50cnkucHk=)
 | `87.01% <100%> (-2.03%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...rflow/providers/apache/cassandra/sensors/record.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3JlY29yZC5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...irflow/providers/apache/cassandra/sensors/table.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3RhYmxlLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==)
 | `0% <0%> (-93.97%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==)
 | `0% <0%> (-91.94%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | ... and [962 
more](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=footer). 
Last update 
[c1ede4d...2f15780](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process

codecov-io edited a comment on issue #7232: [AIRFLOW-6569] Flush pending Sentry 
exceptions before exiting forked process
URL: https://github.com/apache/airflow/pull/7232#issuecomment-581760360
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=h1) 
Report
   > Merging 
[#7232](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c1ede4d63f6498d307ca5d0572296fc99d2eb6a2?src=pr=desc)
 will **increase** coverage by `0.77%`.
   > The diff coverage is `33.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7232/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7232  +/-   ##
   ==
   + Coverage   85.28%   86.05%   +0.77% 
   ==
 Files 761  871 +110 
 Lines   4000440634 +630 
   ==
   + Hits3411634969 +853 
   + Misses   5888 5665 -223
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `64.06% <0%> (-3.16%)` | :arrow_down: |
   | 
[airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9zZW50cnkucHk=)
 | `87.01% <100%> (-2.03%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...rflow/providers/apache/cassandra/sensors/record.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3JlY29yZC5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...irflow/providers/apache/cassandra/sensors/table.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3RhYmxlLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==)
 | `0% <0%> (-93.97%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==)
 | `0% <0%> (-91.94%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | ... and [962 
more](https://codecov.io/gh/apache/airflow/pull/7232/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=footer). 
Last update 
[c1ede4d...2f15780](https://codecov.io/gh/apache/airflow/pull/7232?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mikeclarke commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process

mikeclarke commented on a change in pull request #7232: [AIRFLOW-6569] Flush 
pending Sentry exceptions before exiting forked process
URL: https://github.com/apache/airflow/pull/7232#discussion_r374482129
 
 

 ##
 File path: airflow/task/task_runner/standard_task_runner.py
 ##
 @@ -80,8 +81,11 @@ def _start_by_fork(self):
 
 try:
 args.func(args, dag=self.dag)
+# Explicitly flush any pending exception to Sentry if enabled
+Sentry.flush()
 os._exit(0)
 except Exception:
+Sentry.flush()
 os._exit(1)
 
 Review comment:
   great suggestion, will update - thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates

pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-581749094
 
 
   > You mentioned some DAGs that are usually scheduled daily at a specific 
time, but need to be triggered earlier than the scheduled time once every week 
regularly. For complicated triggering logic like this, did you consider making 
another DAG that triggers this DAG with TriggerDagRunOperator ? Then the DAG 
itself can be set to schedule_interval=None and relies on another DAG to do the 
triggering. It'll also respect allow_trigger_in_future once 
schedule_interval=None is set.
   
   I was not familiar with that; that would definitely suit the bill but does 
seem a little less nice than handling everything within a single DAG. 
Definitely something we can fallback to if these changes aren't liked.
   
   > Having Airflow silently handling trigger dag with duplicated key sounds a 
bit dangerous to me. But maybe others have different opinions.
   
   True - this is an existing codebase with a lot of existing users; a change 
like this could potentially break things.
   
   @kaxil || @nuclearpinguin || others: thoughts on this? TL;DR of the proposal 
is to (a) make this `allow_trigger_in_future` respected in all DAGs regardless 
of their `scheduled_interval`, and (b) gracefully handle the unique constraint 
violation when inserting a DAG run. Thanks!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6273) databricks_hook.py changed how host is pulled



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029573#comment-17029573
 ] 

ASF GitHub Bot commented on AIRFLOW-6273:
-

maxirus commented on pull request #7357: [AIRFLOW-6273] Fixes Databricks 
Connection when using Token
URL: https://github.com/apache/airflow/pull/7357
 
 
   Fixes the Databricks connection when using an Auth Token. Token can be 
stored in a password and follows the general connection schema.
   
   Also fixes [AIRFLOW-6174](https://issues.apache.org/jira/browse/AIRFLOW-6174)
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x ] Description above provides context of the change
   - [x ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x ] Unit tests coverage for changes (not needed for documentation changes)
   - [x ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x ] Relevant documentation is updated including usage instructions.
   - [ x ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> databricks_hook.py changed how host is pulled
> -
>
> Key: AIRFLOW-6273
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6273
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.6
>Reporter: Ameer Elkordy
>Priority: Major
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> AIRFLOW-4923 made a change to how host is pulled based on whether tokens are 
> used or not.  The change results in the host no longer being pulled from the 
> UI host field.  Current work around is to add host to the host field AND to 
> add host to the extra_dejson, which is not intuitive.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] maxirus opened a new pull request #7357: [AIRFLOW-6273] Fixes Databricks Connection when using Token

maxirus opened a new pull request #7357: [AIRFLOW-6273] Fixes Databricks 
Connection when using Token
URL: https://github.com/apache/airflow/pull/7357
 
 
   Fixes the Databricks connection when using an Auth Token. Token can be 
stored in a password and follows the general connection schema.
   
   Also fixes [AIRFLOW-6174](https://issues.apache.org/jira/browse/AIRFLOW-6174)
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x ] Description above provides context of the change
   - [x ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x ] Unit tests coverage for changes (not needed for documentation changes)
   - [x ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x ] Relevant documentation is updated including usage instructions.
   - [ x ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] maxirus closed pull request #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error

maxirus closed pull request #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read 
time out error
URL: https://github.com/apache/airflow/pull/6643
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] maxirus commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error

maxirus commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read 
time out error
URL: https://github.com/apache/airflow/pull/6643#issuecomment-581732169
 
 
   Sooo the test framework has a high barrier to entry and there doesn't seem 
to be any existing tests for the `KubernetesJobWatcher` class to piggy-back on. 
I attempted to get the Test platform going, reading what documentation there 
is, but I just don't have the time to invest in it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6040) Airflow scheduler with kubernetes executor fails :- Unknown error in KubernetesJobWatcher



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029548#comment-17029548
 ] 

ASF GitHub Bot commented on AIRFLOW-6040:
-

maxirus commented on pull request #6643: [AIRFLOW-6040] Fix 
KubernetesJobWatcher Read time out error
URL: https://github.com/apache/airflow/pull/6643
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Airflow scheduler with kubernetes executor fails :- Unknown error in 
> KubernetesJobWatcher
> -
>
> Key: AIRFLOW-6040
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6040
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, executor-kubernetes, scheduler
>Affects Versions: 1.10.6
>Reporter: Ashutosh Srivastava
>Assignee: Tom Vendetta
>Priority: Major
>
> I am trying to set up airflow with the kubernetes executor. I have cloned 
> airflow 1.10.6 and am building the docker image and then deploying it with 
> kube. The pods are running, the service airflow also starts. The webserver is 
> working fine. But when I check the logs for the scheduler I get the following 
> error.
>  
> {{ERROR - Error while health checking kube watcher process. Process died for 
> unknown reasons
> INFO - Event: and now my watch begins starting at resource_version: 0
> ERROR - Unknown error in KubernetesJobWatcher. Failing
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/contrib/executors/kubernetes_executor.py",
>  line 333, in run
> self.worker_uuid, self.kube_config)
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/contrib/executors/kubernetes_executor.py",
>  line 358, in _run
> **kwargs):
>   File "/usr/local/lib/python2.7/dist-packages/kubernetes/watch/watch.py", 
> line 144, in stream
> for line in iter_resp_lines(resp):
>   File "/usr/local/lib/python2.7/dist-packages/kubernetes/watch/watch.py", 
> line 48, in iter_resp_lines
> for seg in resp.read_chunked(decode_content=False):
>   File "/usr/local/lib/python2.7/dist-packages/urllib3/response.py", line 
> 781, in read_chunked
> self._original_response.close()
>   File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
> self.gen.throw(type, value, traceback)
>   File "/usr/local/lib/python2.7/dist-packages/urllib3/response.py", line 
> 439, in _error_catcher
> raise ReadTimeoutError(self._pool, None, "Read timed out.")
> ReadTimeoutError: HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed 
> out.}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] thealmightygrant commented on issue #5813: [AIRFLOW-4526] KubernetesPodOperator gets stuck in Running state when get_logs is set to True and there is a long gap without logs from po

thealmightygrant commented on issue #5813: [AIRFLOW-4526] KubernetesPodOperator 
gets stuck in Running state when get_logs is set to True and there is a long 
gap without logs from pod
URL: https://github.com/apache/airflow/pull/5813#issuecomment-581719105
 
 
   It's unlikely that I'll get back to this anytime soon, but the explanation 
is above and the code should not require a large change. Let me know if one of 
you wants to do this, and I am available to help.
   
   I took a quick look at the latest codebase, and this operator has moved to 
`airflow/providers/cncf/kubernetes/operators/kubernetes_pod`. If no one takes 
this on in  the next few weeks, I'll circle back to it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates

yuqian90 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-581713200
 
 
   > @yuqian90 I see your point - since our use case is a one-off, this extra 
communication of a failed pipeline could be okay. However, for others I wonder 
if this might not be a one-off case; it might be something a team does once a 
week, or once a month, or every time a particular ticket comes in, etc. The 
extra step of handling the failure at re-triggering time could be burdensome 
and yet another thing that goes into an oncall handbook or something that 
causes an oncall page.
   > 
   > Just to clarify as well - the suggestion isn't to make 
`allow_trigger_in_future` be _enabled_ for all DAGs, the suggestion is to make 
`allow_trigger_in_future` _respected_ by all DAGs.
   > 
   > An alternative to changing the default behavior of the scheduler is adding 
another configuration option, something like 
`handle_duplicate_runs_gracefully`, that would allow for handling the 
re-triggering scenario for those that opt-in. My naive feeling is that this 
isn't a very intuitive option and would be difficult to explain, but open to 
alternatives as well.
   > 
   > Thoughts?
   
   You mentioned some DAGs that are usually scheduled daily at a specific time, 
but need to be triggered earlier than the scheduled time once every week 
regularly. For complicated triggering logic like this, did you consider making 
another DAG that triggers this DAG with `TriggerDagRunOperator` ? Then the DAG 
itself can be set to `schedule_interval=None` and relies on another DAG to do 
the triggering. It'll also respect `allow_trigger_in_future` once 
`schedule_interval=None` is set.
   
   Having Airflow silently handling trigger dag with duplicated key sounds a 
bit dangerous to me. But maybe others have different opinions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates

pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-581709034
 
 
   @yuqian90 I see your point - since our use case is a one-off, this extra 
communication of a failed pipeline could be okay. However, for others I wonder 
if this might not be a one-off case; it might be something a team does once a 
week, or once a month, or every time a particular ticket comes in, etc. The 
extra step of handling the failure at re-triggering time could be burdensome 
and yet another thing that goes into an oncall handbook or something that 
causes an oncall page.
   
   Just to clarify as well - the suggestion isn't to make 
`allow_trigger_in_future` be _enabled_ for all DAGs, the suggestion is to make 
`allow_trigger_in_future` *respected* by all DAGs.
   
   An alternative to changing the default behavior of the scheduler is adding 
another configuration option, something like 
`handle_duplicate_runs_gracefully`, that would allow for handling the 
re-triggering scenario for those that opt-in. My naive feeling is that this 
isn't a very intuitive option and would be difficult to explain, but open to 
alternatives as well.
   
   Thoughts?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ryw commented on issue #7351: [AIRFLOW-XXXX] Add scheduler in production section to best practices doc

ryw commented on issue #7351: [AIRFLOW-] Add scheduler in production 
section to best practices doc
URL: https://github.com/apache/airflow/pull/7351#issuecomment-581704942
 
 
   @kaxil hmm looks like random test fail?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates

yuqian90 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-581703874
 
 
   @pbotros if we turn on allow_trigger_in_future to all DAGs (i.e. make it the 
default), and we trigger the DAG for a future date because we are inpatient, it 
probably makes more sense for the scheduler to raise an error when it tries to 
re-trigger at the original scheduled time?
   
   What you want to do sounds like a one-off scenario that can be dealt with 
via communication instead of letting the scheduler handle it silently?
   
   However, if you constantly want to trigger something earlier than it's 
original schedule, is there any change that can be made to the system? E..g why 
not consider changing to external trigger altogether so you can have full 
control over what time the DAG is triggered?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates

pbotros commented on issue #7038: [AIRFLOW-4495] Allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-581698426
 
 
   Sorry for the comment spam on a merged PR, but a quick follow-up to @kaxil's 
comment above:
   
   > DAG is scheduled to run every day at 5 PM
   > 1. Dag is triggered for a future date (ex: 10 Jan 2020 17:00:00) and 
succeeds
   > 2. 10 Jan arrives, Scheduler tries to run it but fails because it violates 
unique key constraint
   
   What if this is the use case we _want_ to support? i.e.: DAG is scheduled to 
run every day at 5 PM, and let's say I did some upstream data changes. I'm 
impatient and want to run the DAG at 4pm for one particular day instead of 5pm. 
Ideally:
   
   - Trigger dag with a future date (ex: 3 Feb 2020 17:00:00) at 4pm via 
airflow CLI; succeeds
   - 5pm arrives, and the scheduler handles the unique key constraint 
gracefully and does not run the DAG
   
   @kaxil @yuqian90 et al, thoughts on the above? Said another way, the 
proposal is to extend `allow_trigger_in_future` to all DAGs, including ones 
with a `schedule_interval`, by handling the unique constraint gracefully.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric

codecov-io edited a comment on issue #6342: [AIRFLOW-5662] fix incorrect naming 
for scheduler used slot metric
URL: https://github.com/apache/airflow/pull/6342#issuecomment-547121627
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=h1) 
Report
   > Merging 
[#6342](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/5f7acb1f0df46794bf63a5cb4794d4cefaa0c391?src=pr=desc)
 will **decrease** coverage by `0.27%`.
   > The diff coverage is `95.23%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6342/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6342  +/-   ##
   ==
   - Coverage   86.35%   86.07%   -0.28% 
   ==
 Files 871  871  
 Lines   4062740655  +28 
   ==
   - Hits3508334994  -89 
   - Misses   5544 5661 +117
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `96.55% <95%> (-0.82%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `89.46% <95.23%> (+0.12%)` | :arrow_up: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `76.08% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [5 
more](https://codecov.io/gh/apache/airflow/pull/6342/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=footer). 
Last update 
[5f7acb1...6fd7ac6](https://codecov.io/gh/apache/airflow/pull/6342?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] youngyjd closed pull request #7356: Merge upstream 20 02 03

youngyjd closed pull request #7356: Merge upstream 20 02 03
URL: https://github.com/apache/airflow/pull/7356
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] youngyjd opened a new pull request #7356: Merge upstream 20 02 03

youngyjd opened a new pull request #7356: Merge upstream 20 02 03
URL: https://github.com/apache/airflow/pull/7356
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6728) Recent Tasks not loading

2020-02-03 Thread Matthew (Jira)

Matthew created AIRFLOW-6728:


 Summary: Recent Tasks not loading
 Key: AIRFLOW-6728
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6728
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui, webserver
Affects Versions: 1.10.7
Reporter: Matthew
Assignee: Matthew


The "Recent Tasks" column of the webserver UI is blank on our production 
instance. This appears to be the result of the `/task_stats` request being too 
large: [https://ookla.d.pr/L5qleA]



While I can't share the actual URL, the math would indicate this anyway. It 
appears that the URL uses a `dag_ids` parameter that is a comma separated list 
in the GET request. Since DAGs can have a `dag_id` length of [250 
characters|[https://github.com/apache/airflow/blob/master/airflow/models/base.py#L35]]
 and the UI will show up to 100 DAGs per page, this means that the URL 
parameters alone could be 25,000 characters (not including domain, etc.). It 
seems that it might be best to switch this request over to POST to accommodate 
the potentially large request body.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ryw commented on issue #7351: [AIRFLOW-XXXX] Add scheduler in production section to best practices doc

ryw commented on issue #7351: [AIRFLOW-] Add scheduler in production 
section to best practices doc
URL: https://github.com/apache/airflow/pull/7351#issuecomment-581684842
 
 
   @kaxil thanks for suggestions - i accepted your changes, then merged back 
into one commit + forced pushed to my branch. is that the right way?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] sarah-johnson commented on issue #5813: [AIRFLOW-4526] KubernetesPodOperator gets stuck in Running state when get_logs is set to True and there is a long gap without logs from pod

sarah-johnson commented on issue #5813: [AIRFLOW-4526] KubernetesPodOperator 
gets stuck in Running state when get_logs is set to True and there is a long 
gap without logs from pod
URL: https://github.com/apache/airflow/pull/5813#issuecomment-581670109
 
 
   I am also running into this issue and would be glad to help on getting a fix 
released.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7133: [AIRFLOW-6535] add exception to fail without retry

codecov-io edited a comment on issue #7133: [AIRFLOW-6535] add exception to 
fail without retry
URL: https://github.com/apache/airflow/pull/7133#issuecomment-573236471
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=h1) 
Report
   > Merging 
[#7133](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/974a87c3872b4c4f8ef74bc385553677d74911ab?src=pr=desc)
 will **increase** coverage by `0.26%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7133/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7133  +/-   ##
   ==
   + Coverage   85.78%   86.05%   +0.26% 
   ==
 Files 866  871   +5 
 Lines   4056440635  +71 
   ==
   + Hits3479834967 +169 
   + Misses   5766 5668  -98
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[...rflow/contrib/hooks/gcp\_video\_intelligence\_hook.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF92aWRlb19pbnRlbGxpZ2VuY2VfaG9vay5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[...irflow/example\_dags/example\_kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5)
 | `85% <ø> (ø)` | :arrow_up: |
   | 
[...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5)
 | `75% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/utils/weekday.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3V0aWxzL3dlZWtkYXkucHk=)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/sensors/qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcXVib2xlX3NlbnNvci5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/utils/gcp\_field\_sanitizer.py](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3V0aWxzL2djcF9maWVsZF9zYW5pdGl6ZXIucHk=)
 | `0% <ø> (ø)` | :arrow_up: |
   | ... and [823 
more](https://codecov.io/gh/apache/airflow/pull/7133/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=footer). 
Last update 
[974a87c...aa011a1](https://codecov.io/gh/apache/airflow/pull/7133?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7355: [AIRFLOW-6727] Fix minor bugs in Release Management scripts

codecov-io edited a comment on issue #7355: [AIRFLOW-6727] Fix minor bugs in 
Release Management scripts
URL: https://github.com/apache/airflow/pull/7355#issuecomment-581662266
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@96f8343`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7355/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7355   +/-   ##
   =
 Coverage  ?   86.06%   
   =
 Files ?  871   
 Lines ?40627   
 Branches  ?0   
   =
 Hits  ?34966   
 Misses? 5661   
 Partials  ?0
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=footer). 
Last update 
[96f8343...6bf4f39](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7355: [AIRFLOW-6727] Fix minor bugs in Release Management scripts

codecov-io commented on issue #7355: [AIRFLOW-6727] Fix minor bugs in Release 
Management scripts
URL: https://github.com/apache/airflow/pull/7355#issuecomment-581662266
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@96f8343`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7355/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7355   +/-   ##
   =
 Coverage  ?   85.88%   
   =
 Files ?  871   
 Lines ?40627   
 Branches  ?0   
   =
 Hits  ?34894   
 Misses? 5733   
 Partials  ?0
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=footer). 
Last update 
[96f8343...6bf4f39](https://codecov.io/gh/apache/airflow/pull/7355?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7351: [AIRFLOW-XXXX] Add scheduler in production section to best practices doc

kaxil commented on a change in pull request #7351: [AIRFLOW-] Add scheduler 
in production section to best practices doc
URL: https://github.com/apache/airflow/pull/7351#discussion_r374380505
 
 

 ##
 File path: docs/best-practices.rst
 ##
 @@ -315,3 +315,16 @@ Some configurations such as Airflow Backend connection 
URI can be derived from b
 .. code::
 
  sql_alchemy_conn_cmd = bash_command_to_run
+
+
+Scheduler Uptime
+-
+
+Airflow users have for a long time been affected by a
+`core Airflow bug `_
+that causes the scheduler to hang without a trace.
+
+Until fully resolved, you can mitigate a few ways:
+
+* Set a reasonable run_duration setting in your `airflow.cfg`. `See example 
`_.
 
 Review comment:
   ```suggestion
   * Set a reasonable run_duration setting in your ``airflow.cfg``. `See 
example 
`_.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7351: [AIRFLOW-XXXX] Add scheduler in production section to best practices doc

kaxil commented on a change in pull request #7351: [AIRFLOW-] Add scheduler 
in production section to best practices doc
URL: https://github.com/apache/airflow/pull/7351#discussion_r374380633
 
 

 ##
 File path: docs/best-practices.rst
 ##
 @@ -315,3 +315,16 @@ Some configurations such as Airflow Backend connection 
URI can be derived from b
 .. code::
 
  sql_alchemy_conn_cmd = bash_command_to_run
+
+
+Scheduler Uptime
+-
+
+Airflow users have for a long time been affected by a
+`core Airflow bug `_
+that causes the scheduler to hang without a trace.
+
+Until fully resolved, you can mitigate a few ways:
+
+* Set a reasonable run_duration setting in your `airflow.cfg`. `See example 
`_.
+* Add an `exec` style health check to your helm charts on the scheduler 
deployment to fail if the scheduler has not heartbeat in a while. `See example 
`_.
 
 Review comment:
   ```suggestion
   * Add an ``exec`` style health check to your helm charts on the scheduler 
deployment to fail if the scheduler has not heartbeat in a while. `See example 
`_.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6630) Resolve Handlebars advisory

2020-02-03 Thread ASF subversion and git services (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-6630.
-
Resolution: Fixed

> Resolve Handlebars advisory 
> 
>
> Key: AIRFLOW-6630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6630
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.9
>
>
> Security scan turned up Handlebars 4.3.5 as a vulnerability per this NPM 
> advisory [https://www.npmjs.com/advisories/1324] - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6630) Resolve Handlebars advisory



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029348#comment-17029348
 ] 

ASF subversion and git services commented on AIRFLOW-6630:
--

Commit 5f7acb1f0df46794bf63a5cb4794d4cefaa0c391 in airflow's branch 
refs/heads/master from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5f7acb1 ]

[AIRFLOW-6630] Resolve handlebars advisory (#7284)



> Resolve Handlebars advisory 
> 
>
> Key: AIRFLOW-6630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6630
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.9
>
>
> Security scan turned up Handlebars 4.3.5 as a vulnerability per this NPM 
> advisory [https://www.npmjs.com/advisories/1324] - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6630) Resolve Handlebars advisory



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029347#comment-17029347
 ] 

ASF GitHub Bot commented on AIRFLOW-6630:
-

kaxil commented on pull request #7284: [AIRFLOW-6630] Bump handlebars to 
resolve handlebars advisory
URL: https://github.com/apache/airflow/pull/7284
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Resolve Handlebars advisory 
> 
>
> Key: AIRFLOW-6630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6630
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.9
>
>
> Security scan turned up Handlebars 4.3.5 as a vulnerability per this NPM 
> advisory [https://www.npmjs.com/advisories/1324] - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil merged pull request #7284: [AIRFLOW-6630] Bump handlebars to resolve handlebars advisory

kaxil merged pull request #7284: [AIRFLOW-6630] Bump handlebars to resolve 
handlebars advisory
URL: https://github.com/apache/airflow/pull/7284
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7353: [AIRFLOW-6685] Data Quality Check operators

mik-laj commented on a change in pull request #7353: [AIRFLOW-6685] Data 
Quality Check operators
URL: https://github.com/apache/airflow/pull/7353#discussion_r374378103
 
 

 ##
 File path: airflow/operators/data_quality_check.py
 ##
 @@ -0,0 +1,207 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.utils.decorators import apply_defaults
+from airflow.models import BaseOperator
+from airflow.hooks.base_hook import BaseHook
+from airflow.hooks.postgres_hook import PostgresHook
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.hooks.hive_hooks import HiveServer2Hook
+from airflow import AirflowException
+
+class BaseDataQualityOperator(BaseOperator):
+"""
+BaseDataQualityOperator is an abstract base operator class to
+perform data quality checks
+
+:param sql: sql (or path to sql) code to be executed
+:type sql: str
+:param conn_type: database type
+:type conn_type: str
+:param conn_id: connection id of database
+:type conn_id: str
+:param push_conn_type: (optional) external database type
+:type push_conn_type: str
+:param push_conn_id: (optional) connection id of external database
+:type push_conn_id: str
+:param check_description: (optional) description of data quality sql 
statement
+:type check_description: str
+"""
+
+template_fields = ('sql',)
+template_ext = ('.sql',)
+
+@apply_defaults
+def __init__(self,
+ sql,
+ conn_id,
+ push_conn_id=None,
+ check_description=None,
+ *args,
+ **kwargs
+ ):
+super().__init__(*args, **kwargs)
+self.conn_id = conn_id
+self.push_conn_id = push_conn_id
+self.sql = sql
+self.check_description = check_description
+
+def execute(self, context):
+"""Method where data quality check is performed """
+raise NotImplementedError
+
+def push(self, info_dict):
+"""Send data check info and metadata to an external database."""
+pass
+
+def send_failure_notification(self, info_dict):
+"""
+send_failure_notification will throw an AirflowException with logging
+information and dq check results from the failed task that was just 
run.
+"""
+body = f"""Data Quality Check: "{info_dict.get("task_id")}" failed.
+DAG: {self.dag_id}
+Task_id: {info_dict.get("task_id")}
+Check description: {info_dict.get("description")}
+Execution date: {info_dict.get("execution_date")}
+SQL: {self.sql}
+Result: {round(info_dict.get("result"), 2)} is not within thresholds 
{info_dict.get("min_threshold")} and {info_dict.get("max_threshold")}"""
+raise AirflowException(body)
+
+def _get_hook(self, conn_id):
+"""
+_get_hook is a helper function for get_sql_value. Returns a database
+hook depending on the conn_type and conn_id specified. Method will 
raise
+an exception if hook is not supported.
+"""
+
+conn_type = BaseHook.get_connection(conn_id).conn_type
 
 Review comment:
   Why didn't you use the Connection.get_hook method?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6727) Fix minor issues in Release Management scripts



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029338#comment-17029338
 ] 

ASF GitHub Bot commented on AIRFLOW-6727:
-

kaxil commented on pull request #7355: [AIRFLOW-6727] Fix minor bugs in Release 
Management scripts
URL: https://github.com/apache/airflow/pull/7355
 
 
   There are some minor issues:
   
   Click needs to be >= 7.0
   Path of CHANGELOG.txt
   Path of template files to send automated emails
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix minor issues in Release Management scripts
> --
>
> Key: AIRFLOW-6727
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6727
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0, 1.10.9
>
>
> There are some minor issues:
> * Click needs to be >= 7.0
> * Path of CHANGELOG.txt
> * Path of template files to send automated emails



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6727) Fix minor issues in Release Management scripts



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6727:

Fix Version/s: 1.10.9

> Fix minor issues in Release Management scripts
> --
>
> Key: AIRFLOW-6727
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6727
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0, 1.10.9
>
>
> There are some minor issues:
> * Click needs to be >= 7.0
> * Path of CHANGELOG.txt
> * Path of template files to send automated emails



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7355: [AIRFLOW-6727] Fix minor bugs in Release Management scripts

kaxil opened a new pull request #7355: [AIRFLOW-6727] Fix minor bugs in Release 
Management scripts
URL: https://github.com/apache/airflow/pull/7355
 
 
   There are some minor issues:
   
   Click needs to be >= 7.0
   Path of CHANGELOG.txt
   Path of template files to send automated emails
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6727) Fix minor issues in Release Management scripts

Kaxil Naik created AIRFLOW-6727:
---

 Summary: Fix minor issues in Release Management scripts
 Key: AIRFLOW-6727
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6727
 Project: Apache Airflow
  Issue Type: Bug
  Components: PR tool
Affects Versions: 2.0.0
Reporter: Kaxil Naik
Assignee: Kaxil Naik
 Fix For: 2.0.0


There are some minor issues:
* Click needs to be >= 7.0
* Path of CHANGELOG.txt
* Path of template files to send automated emails



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7354: [AIRFLOW-XXXX] Add versions_added field to configs

kaxil opened a new pull request #7354: [AIRFLOW-] Add versions_added field 
to configs
URL: https://github.com/apache/airflow/pull/7354
 
 
   Add versions_added field to configs until 1.10.1
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow

stale[bot] commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to 
trigger job from Airflow
URL: https://github.com/apache/airflow/pull/2708#issuecomment-581641177
 
 
   This issue has been automatically marked as stale because it has not had 
recent activity. It will be closed if no further activity occurs. Thank you for 
your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6685) Add Data Quality Operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029324#comment-17029324
 ] 

ASF GitHub Bot commented on AIRFLOW-6685:
-

alexzlue commented on pull request #7353: [AIRFLOW-6685] Data Quality Check 
operators
URL: https://github.com/apache/airflow/pull/7353
 
 
   This PR includes 3 operators:
   `BaseDataQualityOperator`
   - contains shared attributes and methods that data quality check operators 
utilize
   a base class that can be used to create other dq operators
   
   `DataQualityThresholdCheckOperator`
   - will check a single value, sql result against a threshold range, and will 
fail a task if it is outside this range.
   
   `DataQulaityThresholdSQLCheckOperator`
   - Similar to DataQualityThresholdCheckOperator, but thresholds are 
sql-evaluated values, for dynamic threshold ranging.
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Data Quality Operators 
> ---
>
> Key: AIRFLOW-6685
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6685
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: alex l
>Assignee: alex l
>Priority: Major
>
> Add Data Quality Operators to improve data quality testing on data 
> workflows/pipelines. This includes 3 operators:
>  * BaseDataQualityOperator
>  ** contains shared attributes and methods that data quality check operators 
> utilize
>  ** a base class that can be used to create other dq operators
>  * DataQualityThresholdCheckOperator
>  ** will check a single value, sql result against a threshold range, and will 
> fail a task if it is outside this range.
>  * DataQulaityThresholdSQLCheckOperator
>  ** Similar to DataQualityThresholdCheckOperator, but thresholds are 
> sql-evaluated values, for dynamic threshold ranging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7353: [AIRFLOW-6685] Data Quality Check operators

boring-cyborg[bot] commented on issue #7353: [AIRFLOW-6685] Data Quality Check
operators
URL: https://github.com/apache/airflow/pull/7353#issuecomment-581637170

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for
testing locally, it’s a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] alexzlue opened a new pull request #7353: [AIRFLOW-6685] Data Quality Check operators

2020-02-03 Thread ASF subversion and git services (Jira)

alexzlue opened a new pull request #7353: [AIRFLOW-6685] Data Quality Check 
operators
URL: https://github.com/apache/airflow/pull/7353
 
 
   This PR includes 3 operators:
   `BaseDataQualityOperator`
   - contains shared attributes and methods that data quality check operators 
utilize
   a base class that can be used to create other dq operators
   
   `DataQualityThresholdCheckOperator`
   - will check a single value, sql result against a threshold range, and will 
fail a task if it is outside this range.
   
   `DataQulaityThresholdSQLCheckOperator`
   - Similar to DataQualityThresholdCheckOperator, but thresholds are 
sql-evaluated values, for dynamic threshold ranging.
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029306#comment-17029306
 ] 

ASF subversion and git services commented on AIRFLOW-4495:
--

Commit 675c7a4139518bc01b7bbac5d25896d3f12d4752 in airflow's branch 
refs/heads/v1-10-stable from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=675c7a4 ]

[AIRFLOW-4495] Allow externally triggered dags to run for future exec dates 
(#7038)

(cherry-picked from 6414d38a0)


> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> 1. 
>  useful to handle future date for externally triggered batch process where 
> ingesting 'forecast' data where filename date is in the future
> 2.
>  this error is just in the scheduler log and not propagated up, so the dag 
> stays in 'running' state forever (or for 1 year waiting for the time to pass 
> :) )
>  ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>  
>  
> fix below works if u only have externally triggered DAGs:
>  
> commenting below ti_deps\deps\runnable_exec_date_dep.py
>     #if ti.execution_date > cur_date:
>     #    yield self._failing_status(
>     #    reason="Execution date \{0} is in the future (the current "
>     #   "date is \{1}).".format(ti.execution_date.isoformat(),
>     #  cur_date.isoformat()))
>  
> commenting below jobs.py
>     # don't consider runs that are executed in the future
>     #if run.execution_date > timezone.utcnow():
>     #    self.log.error(
>     #    "Execution date is in future: %s",
>     #    run.execution_date
>     #    )
>     #    continue
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6666) Resolve js-yaml advisories

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029310#comment-17029310
 ] 

ASF subversion and git services commented on AIRFLOW-:
--

Commit caa356713ada4b58952747ca7984ac5f9605468d in airflow's branch 
refs/heads/v1-10-stable from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=caa3567 ]

[AIRFLOW-] Resolve js-yaml advisory (#7283)

(cherry-picked from 17464bdbe)


> Resolve js-yaml advisories
> --
>
> Key: AIRFLOW-
> URL: https://issues.apache.org/jira/browse/AIRFLOW-
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> Discovered via `npm audit`
>  # https://npmjs.com/advisories/788 
>  # https://npmjs.com/advisories/813    



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6683) Make API respect serialized DAG feature

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029305#comment-17029305
 ] 

ASF subversion and git services commented on AIRFLOW-6683:
--

Commit f9648013c7a633c49d2b17c640a093ae51eadd8c in airflow's branch 
refs/heads/v1-10-stable from Anita Fronczak
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f964801 ]

[AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296)

Make REST API respect core.store_serialized_dags setting

(cherry picked from commit eb0f6d1b2c317088ee261aa10caa6df783fb8652)


> Make API respect serialized DAG feature
> ---
>
> Key: AIRFLOW-6683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6683
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, serialization
>Affects Versions: 1.10.7
>Reporter: Anita Fronczak
>Assignee: Anita Fronczak
>Priority: Major
> Fix For: 1.10.8
>
>
> Currently, when running a DAG using HTTP request STORE_ERIALIZED_DAG setting 
> is not respected.
> Example call:
> curl -X POST \ [http://localhost:8080/api/experimental/dags/DAG77/dag_runs] \ 
> -H 'Cache-Control: no-cache' \ -H 'Content-Type: application/json' \ -d 
> '\{"conf":"{\"key\":\"value\"}"}'
> Here is a PR that fixes it: [https://github.com/apache/airflow/pull/7296]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4026) Add DAG Labels to improve filtering of DAGs

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029313#comment-17029313
 ] 

ASF subversion and git services commented on AIRFLOW-4026:
--

Commit 6f4e3b6a05dd4e6944fefa7f6fdada64b5c6e025 in airflow's branch 
refs/heads/v1-10-stable from Zacharya
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6f4e3b6 ]

[AIRFLOW-4026] Add filter by DAG tags (#6489)

(cherry-picked from a7cacf593)


> Add DAG Labels to improve filtering of DAGs
> ---
>
> Key: AIRFLOW-4026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4026
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0, 1.10.8
>
> Attachments: Screen Shot 2019-03-06 at 09.39.22.png
>
>
>  !Screen Shot 2019-03-06 at 09.39.22.png! 
> See Discussion: 
> https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551861675239000



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6438) blocked end point returns all dags by default

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029312#comment-17029312
 ] 

ASF subversion and git services commented on AIRFLOW-6438:
--

Commit b461e831c710212defa32687d8c49631acc7f58a in airflow's branch 
refs/heads/v1-10-stable from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b461e83 ]

[AIRFLOW-6438] Filter DAGs returned by blocked (#7019)

(cherry-picked from 5491f4d40413075d85b50f39c27ad871741c68d8)


> blocked end point returns all dags by default
> -
>
> Key: AIRFLOW-6438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6438
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> The /blocked end point returns any blocked dags in the entire system by 
> default. This can be quite slow when running a large amount of dags (2500+ in 
> our case).
> The accompanying pull request adds a dag_ids get parameter to the blocked end 
> point which is populated by the dags present on the page.
> Please see previous issues which fixed the other end points in a similar 
> manner AIRFLOW-6239, AIRFLOW-6238, AIRFLOW-6095



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6632) Resolve lodash advisory

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029309#comment-17029309
 ] 

ASF subversion and git services commented on AIRFLOW-6632:
--

Commit ad4192b921cfa3e97b844f78239ad6bd13c74ea3 in airflow's branch 
refs/heads/v1-10-stable from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ad4192b ]

[AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)

(cherry-picked from 6dc9d9fa5)


> Resolve lodash advisory
> ---
>
> Key: AIRFLOW-6632
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6632
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> https://npmjs.com/advisories/577   
> Security scan turned up lodash 4.17.5 as a vulnerability per this NPM 
> advisory  - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6667) Resolve serialize-javascript advisory

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029308#comment-17029308
 ] 

ASF subversion and git services commented on AIRFLOW-6667:
--

Commit 18ea933acd2d4ebed505575465f6c4c5a323313f in airflow's branch 
refs/heads/v1-10-stable from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=18ea933 ]

[AIRFLOW-6667] Resolve serialize-javascript advisory (#7282)

(cherry-picked from 9742a0c2bb25d506ded3146dc8f381063866a3ee)


> Resolve serialize-javascript advisory
> -
>
> Key: AIRFLOW-6667
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6667
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> Discovered w/ `npm audit`:
>  * https://npmjs.com/advisories/1426



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6686) Syntax error constructing list of process ids

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029307#comment-17029307
 ] 

ASF subversion and git services commented on AIRFLOW-6686:
--

Commit add7acd24d71f8b327e7f16d558a7a7ad915a508 in airflow's branch 
refs/heads/v1-10-stable from Oluwafemi Sule
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=add7acd ]

[AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)

Construction of list of pids passed in kill command raises a Syntax error 
because
ordering of the arguments to `map` function doesn't conform to function
parameter definition.

A list comprehension replaces the existing `map` function to make the code
forward compatible with Python 3 and at the same time expand to a list of pids.

(cherry picked from commit d1cedc4f691e683bf1ed54aee3a2c1a54013bbb3)


> Syntax error constructing list of process ids
> -
>
> Key: AIRFLOW-6686
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6686
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Affects Versions: 1.10.7
>Reporter: Oluwafemi Sule
>Assignee: Oluwafemi Sule
>Priority: Major
> Fix For: 1.10.8
>
>
> There is Syntax error constructing list of process ids used in process 
> termination command.
> The Syntax error results from wrong ordering of arguments to builtin map 
> function.
> [https://github.com/apache/airflow/blob/master/airflow/utils/helpers.py#L214]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

svn commit: r37855 - /dev/airflow/1.10.8rc1/

2020-02-03 Thread kaxilnaik

Author: kaxilnaik
Date: Mon Feb  3 21:30:50 2020
New Revision: 37855

Log:
Add artifacts for Airflow 1.10.8rc1

Added:
dev/airflow/1.10.8rc1/
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz   (with props)
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.asc
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.sha512
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz   (with props)
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.asc
dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.sha512
dev/airflow/1.10.8rc1/apache_airflow-1.10.8rc1-py2.py3-none-any.whl   (with 
props)
dev/airflow/1.10.8rc1/apache_airflow-1.10.8rc1-py2.py3-none-any.whl.asc
dev/airflow/1.10.8rc1/apache_airflow-1.10.8rc1-py2.py3-none-any.whl.sha512

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz
==
Binary file - no diff available.

Propchange: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.asc
==
--- dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.asc (added)
+++ dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.asc Mon Feb  3 
21:30:50 2020
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+
+iQEzBAABCAAdFiEEEnF1VgQO7y7q8bnCdfzNCiX6DksFAl44ja4ACgkQdfzNCiX6
+Dktyjwf/WqFB9s1FPr5VF/ph7y25n8oWNafTJYFasdIEGbM/srWtYzKO+fEKero7
+cDfy7QrlH9iuEeSzewOprqQo4dVsT2aTFx4opBtFBu02EJZoaRvztcNpSFbViM4O
+zgzaJTJ8sxOyNX2SYoWA4W+YjwGVoOwWHnrQrtsQN9j60M1fWf1Q32pmnMdR0OwJ
+Rd3t2BMUK8btH0DEFC7Nb3FHfJ81viS0kbOHfEhgZUjnEsfF2/RNPDCUzp0Hr7yd
+VFlUD2p0xmBhcZlXe+2nmeRx6Gtt1lkK2AmDYhX6MsOFHYUuR5YgGzDXgT9Z5tJF
+JBrUlZOMHy4hiwL4KKc4ZwwqcdHTog==
+=QmZF
+-END PGP SIGNATURE-

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.sha512
==
--- dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.sha512 (added)
+++ dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-bin.tar.gz.sha512 Mon Feb  3 
21:30:50 2020
@@ -0,0 +1,4 @@
+apache-airflow-1.10.8rc1-bin.tar.gz: 6F1C776C DEC92001 4F320C9B 7FE8E7A0
+ CB77B61D CB3E0716 3914E365 EEEC5F65
+ F13F6F00 357A3C73 0CA22444 29879913
+ 3DA31C4E 130EBE1B 4AC150D5 A20E377B

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz
==
Binary file - no diff available.

Propchange: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.asc
==
--- dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.asc (added)
+++ dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.asc Mon Feb  3 
21:30:50 2020
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+
+iQEzBAABCAAdFiEEEnF1VgQO7y7q8bnCdfzNCiX6DksFAl44jaUACgkQdfzNCiX6
+Dku7qQf+LCGb1JgdutyFRW1s6xH1VTPocufvMhBXJVkh2Mxfkym0tD1GzmfFaBoZ
+qWg0rNaH8VfNsoNxBZq2F5/51x/DdnBXZhtA5QzGlN0OWqrL6MrDyE8dMWYX7JCH
+jNTRmUO/qpOugd3bMBn86eNMU0JbHniJgpc20ORWLSHgZPmgy8BjrE9Vm1zaR4bM
+/sMPC5H+OU9W22qCJHB39stet7uK1IwpMlGkwcCHmcSQ88GmElE9fIi7KgdNZWGg
+j+M8PhEpSv0UQp7LHsJ9JUEu7Je+9QUGVJmv34S+kGrJ8OnrdChrTtwBRB4jFx4n
+ar4kTSrazZduiSOLuzrrCP11R3UtYQ==
+=Qtyo
+-END PGP SIGNATURE-

Added: dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.sha512
==
--- dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.sha512 (added)
+++ dev/airflow/1.10.8rc1/apache-airflow-1.10.8rc1-source.tar.gz.sha512 Mon Feb 
 3 21:30:50 2020
@@ -0,0 +1,4 @@
+apache-airflow-1.10.8rc1-source.tar.gz: 159A457A 851F2E94 BAFE5589 3BA88554
+E40D4165 494946C4 FA0B8109 95E36DA8
+BFBCA8EC E4096F58 F9D11619 1D74B2AB
+8CCA85B9 F6F0239B 093DA8EF 184D5691

Added: dev/airflow/1.10.8rc1/apache_airflow-1.10.8rc1-py2.py3-none-any.whl
==
Binary file - no diff available.

Propchange: dev/airflow/1.10.8rc1/apache_airflow-1.10.8rc1-py2.py3-none-any.whl
--
svn:mime-type = application/octet-stream

Added:

[GitHub] [airflow] dossett commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library

dossett commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to 
use python library
URL: https://github.com/apache/airflow/pull/6371#issuecomment-581625951
 
 
   That link may not expand well.  It's to line 64 in the 
`airflow/gcp/hooks/dataproc.py` that was renamed as part of this change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dossett commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library

2020-02-03 Thread ASF subversion and git services (Jira)

dossett commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to 
use python library
URL: https://github.com/apache/airflow/pull/6371#issuecomment-581624265
 
 
   This seems to have made some substantive changes to the behavior of the 
operators, was that intended?  For example, functionality to detect and 
reattach to a running job is gone.  See the code starting here for 
functionality that wasn't ported. 
https://github.com/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3#diff-0be5a6bccaef6a25d9ff5d63a92a12f0L64-L65
   
   @potiuk @mik-laj @nuclearpinguin


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6686) Syntax error constructing list of process ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029282#comment-17029282
 ] 

ASF subversion and git services commented on AIRFLOW-6686:
--

Commit add7acd24d71f8b327e7f16d558a7a7ad915a508 in airflow's branch 
refs/heads/v1-10-test from Oluwafemi Sule
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=add7acd ]

[AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)

Construction of list of pids passed in kill command raises a Syntax error 
because
ordering of the arguments to `map` function doesn't conform to function
parameter definition.

A list comprehension replaces the existing `map` function to make the code
forward compatible with Python 3 and at the same time expand to a list of pids.

(cherry picked from commit d1cedc4f691e683bf1ed54aee3a2c1a54013bbb3)


> Syntax error constructing list of process ids
> -
>
> Key: AIRFLOW-6686
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6686
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Affects Versions: 1.10.7
>Reporter: Oluwafemi Sule
>Assignee: Oluwafemi Sule
>Priority: Major
> Fix For: 1.10.8
>
>
> There is Syntax error constructing list of process ids used in process 
> termination command.
> The Syntax error results from wrong ordering of arguments to builtin map 
> function.
> [https://github.com/apache/airflow/blob/master/airflow/utils/helpers.py#L214]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6438) blocked end point returns all dags by default

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029286#comment-17029286
 ] 

ASF subversion and git services commented on AIRFLOW-6438:
--

Commit b461e831c710212defa32687d8c49631acc7f58a in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b461e83 ]

[AIRFLOW-6438] Filter DAGs returned by blocked (#7019)

(cherry-picked from 5491f4d40413075d85b50f39c27ad871741c68d8)


> blocked end point returns all dags by default
> -
>
> Key: AIRFLOW-6438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6438
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> The /blocked end point returns any blocked dags in the entire system by 
> default. This can be quite slow when running a large amount of dags (2500+ in 
> our case).
> The accompanying pull request adds a dag_ids get parameter to the blocked end 
> point which is populated by the dags present on the page.
> Please see previous issues which fixed the other end points in a similar 
> manner AIRFLOW-6239, AIRFLOW-6238, AIRFLOW-6095



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6666) Resolve js-yaml advisories

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029285#comment-17029285
 ] 

ASF subversion and git services commented on AIRFLOW-:
--

Commit caa356713ada4b58952747ca7984ac5f9605468d in airflow's branch 
refs/heads/v1-10-test from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=caa3567 ]

[AIRFLOW-] Resolve js-yaml advisory (#7283)

(cherry-picked from 17464bdbe)


> Resolve js-yaml advisories
> --
>
> Key: AIRFLOW-
> URL: https://issues.apache.org/jira/browse/AIRFLOW-
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> Discovered via `npm audit`
>  # https://npmjs.com/advisories/788 
>  # https://npmjs.com/advisories/813    



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4026) Add DAG Labels to improve filtering of DAGs

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029287#comment-17029287
 ] 

ASF subversion and git services commented on AIRFLOW-4026:
--

Commit 6f4e3b6a05dd4e6944fefa7f6fdada64b5c6e025 in airflow's branch 
refs/heads/v1-10-test from Zacharya
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6f4e3b6 ]

[AIRFLOW-4026] Add filter by DAG tags (#6489)

(cherry-picked from a7cacf593)


> Add DAG Labels to improve filtering of DAGs
> ---
>
> Key: AIRFLOW-4026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4026
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0, 1.10.8
>
> Attachments: Screen Shot 2019-03-06 at 09.39.22.png
>
>
>  !Screen Shot 2019-03-06 at 09.39.22.png! 
> See Discussion: 
> https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551861675239000



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6683) Make API respect serialized DAG feature

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029280#comment-17029280
 ] 

ASF subversion and git services commented on AIRFLOW-6683:
--

Commit f9648013c7a633c49d2b17c640a093ae51eadd8c in airflow's branch 
refs/heads/v1-10-test from Anita Fronczak
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f964801 ]

[AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296)

Make REST API respect core.store_serialized_dags setting

(cherry picked from commit eb0f6d1b2c317088ee261aa10caa6df783fb8652)


> Make API respect serialized DAG feature
> ---
>
> Key: AIRFLOW-6683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6683
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, serialization
>Affects Versions: 1.10.7
>Reporter: Anita Fronczak
>Assignee: Anita Fronczak
>Priority: Major
> Fix For: 1.10.8
>
>
> Currently, when running a DAG using HTTP request STORE_ERIALIZED_DAG setting 
> is not respected.
> Example call:
> curl -X POST \ [http://localhost:8080/api/experimental/dags/DAG77/dag_runs] \ 
> -H 'Cache-Control: no-cache' \ -H 'Content-Type: application/json' \ -d 
> '\{"conf":"{\"key\":\"value\"}"}'
> Here is a PR that fixes it: [https://github.com/apache/airflow/pull/7296]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6632) Resolve lodash advisory

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029284#comment-17029284
 ] 

ASF subversion and git services commented on AIRFLOW-6632:
--

Commit ad4192b921cfa3e97b844f78239ad6bd13c74ea3 in airflow's branch 
refs/heads/v1-10-test from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ad4192b ]

[AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)

(cherry-picked from 6dc9d9fa5)


> Resolve lodash advisory
> ---
>
> Key: AIRFLOW-6632
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6632
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> https://npmjs.com/advisories/577   
> Security scan turned up lodash 4.17.5 as a vulnerability per this NPM 
> advisory  - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6667) Resolve serialize-javascript advisory

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029283#comment-17029283
 ] 

ASF subversion and git services commented on AIRFLOW-6667:
--

Commit 18ea933acd2d4ebed505575465f6c4c5a323313f in airflow's branch 
refs/heads/v1-10-test from Ry Walker
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=18ea933 ]

[AIRFLOW-6667] Resolve serialize-javascript advisory (#7282)

(cherry-picked from 9742a0c2bb25d506ded3146dc8f381063866a3ee)


> Resolve serialize-javascript advisory
> -
>
> Key: AIRFLOW-6667
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6667
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.8
>
>
> Discovered w/ `npm audit`:
>  * https://npmjs.com/advisories/1426



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029281#comment-17029281
 ] 

ASF subversion and git services commented on AIRFLOW-4495:
--

Commit 675c7a4139518bc01b7bbac5d25896d3f12d4752 in airflow's branch 
refs/heads/v1-10-test from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=675c7a4 ]

[AIRFLOW-4495] Allow externally triggered dags to run for future exec dates 
(#7038)

(cherry-picked from 6414d38a0)


> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> 1. 
>  useful to handle future date for externally triggered batch process where 
> ingesting 'forecast' data where filename date is in the future
> 2.
>  this error is just in the scheduler log and not propagated up, so the dag 
> stays in 'running' state forever (or for 1 year waiting for the time to pass 
> :) )
>  ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>  
>  
> fix below works if u only have externally triggered DAGs:
>  
> commenting below ti_deps\deps\runnable_exec_date_dep.py
>     #if ti.execution_date > cur_date:
>     #    yield self._failing_status(
>     #    reason="Execution date \{0} is in the future (the current "
>     #   "date is \{1}).".format(ti.execution_date.isoformat(),
>     #  cur_date.isoformat()))
>  
> commenting below jobs.py
>     # don't consider runs that are executed in the future
>     #if run.execution_date > timezone.utcnow():
>     #    self.log.error(
>     #    "Execution date is in future: %s",
>     #    run.execution_date
>     #    )
>     #    continue
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #7347: [AIRFLOW-XXXX] Add pre-commit check for utf-8 file encoding

2020-02-03 Thread ASF subversion and git services (Jira)

potiuk commented on issue #7347: [AIRFLOW-] Add pre-commit check for utf-8 
file encoding
URL: https://github.com/apache/airflow/pull/7347#issuecomment-581619192
 
 
   I think you need to apply the precommit in the next push @feluelle !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6678) KubernetesPodOperator logs pod events on failure



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029268#comment-17029268
 ] 

ASF subversion and git services commented on AIRFLOW-6678:
--

Commit 96f834389e03884025534fabd862155061f53fd0 in airflow's branch 
refs/heads/master from MatthewRBruce
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=96f8343 ]

[AIRFLOW-6678] Pull event logs from Kubernetes (#7292)

Adds an option (defaults to True) to pull and log events from a
Kubernetes pod that fails.

> KubernetesPodOperator logs pod events on failure
> 
>
> Key: AIRFLOW-6678
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6678
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.7
>Reporter: Matthew Bruce
>Assignee: Matthew Bruce
>Priority: Minor
> Fix For: 1.10.8
>
>
> Currently if a pod fails to launch via a KubernetesPodOperator, the events 
> from the pod are not pulled back and logged - i.e. if a pod fails to launch 
> because the target image can't be found, or a volume fails to mount the user 
> gets no indication of this in the airflow logs.
> There should be an option to pull these events back in the case of a pod 
> failure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6678) KubernetesPodOperator logs pod events on failure

2020-02-03 Thread Jarek Potiuk (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6678.
---
Fix Version/s: (was: 2.0.0)
   1.10.8
   Resolution: Fixed

> KubernetesPodOperator logs pod events on failure
> 
>
> Key: AIRFLOW-6678
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6678
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.7
>Reporter: Matthew Bruce
>Assignee: Matthew Bruce
>Priority: Minor
> Fix For: 1.10.8
>
>
> Currently if a pod fails to launch via a KubernetesPodOperator, the events 
> from the pod are not pulled back and logged - i.e. if a pod fails to launch 
> because the target image can't be found, or a volume fails to mount the user 
> gets no indication of this in the airflow logs.
> There should be an option to pull these events back in the case of a pod 
> failure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #7292: [AIRFLOW-6678] Pull event logs from Kubernetes pod on failure

potiuk commented on issue #7292: [AIRFLOW-6678] Pull event logs from Kubernetes 
pod on failure
URL: https://github.com/apache/airflow/pull/7292#issuecomment-581616299
 
 
   Thanks @MatthewRBruce !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6678) KubernetesPodOperator logs pod events on failure



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029267#comment-17029267
 ] 

ASF GitHub Bot commented on AIRFLOW-6678:
-

potiuk commented on pull request #7292: [AIRFLOW-6678] Pull event logs from 
Kubernetes pod on failure
URL: https://github.com/apache/airflow/pull/7292
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> KubernetesPodOperator logs pod events on failure
> 
>
> Key: AIRFLOW-6678
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6678
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.7
>Reporter: Matthew Bruce
>Assignee: Matthew Bruce
>Priority: Minor
> Fix For: 2.0.0
>
>
> Currently if a pod fails to launch via a KubernetesPodOperator, the events 
> from the pod are not pulled back and logged - i.e. if a pod fails to launch 
> because the target image can't be found, or a volume fails to mount the user 
> gets no indication of this in the airflow logs.
> There should be an option to pull these events back in the case of a pod 
> failure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7292: [AIRFLOW-6678] Pull event logs from Kubernetes pod on failure

potiuk merged pull request #7292: [AIRFLOW-6678] Pull event logs from 
Kubernetes pod on failure
URL: https://github.com/apache/airflow/pull/7292
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7292: [AIRFLOW-6678] Pull event logs from Kubernetes pod on failure

boring-cyborg[bot] commented on issue #7292: [AIRFLOW-6678] Pull event logs 
from Kubernetes pod on failure
URL: https://github.com/apache/airflow/pull/7292#issuecomment-581616386
 
 
   Awesome work, congrats on your first merged pull request!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7292: [AIRFLOW-6678] Pull event logs from Kubernetes pod on failure

codecov-io edited a comment on issue #7292: [AIRFLOW-6678] Pull event logs from 
Kubernetes pod on failure
URL: https://github.com/apache/airflow/pull/7292#issuecomment-581455946
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=h1) 
Report
   > Merging 
[#7292](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/a02265382d5c0381c5b8dbd05547bdda493fc6f1?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `71.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7292/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7292  +/-   ##
   ==
   - Coverage   86.35%   86.34%   -0.02% 
   ==
 Files 871  871  
 Lines   4061340627  +14 
   ==
   + Hits3507335079   +6 
   - Misses   5540 5548   +8
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `92.25% <100%> (+0.28%)` | :arrow_up: |
   | 
[...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==)
 | `93.61% <55.55%> (-4.04%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `76.08% <0%> (ø)` | :arrow_up: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==)
 | `77.55% <0%> (ø)` | :arrow_up: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `90.72% <0%> (-1.16%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.73% <0%> (ø)` | :arrow_up: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7292/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `89.34% <0%> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=footer). 
Last update 
[a022653...80866b9](https://codecov.io/gh/apache/airflow/pull/7292?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6855: [AIRFLOW-6307] Improve support for a special characters in DAG's name in stats view

mik-laj commented on a change in pull request #6855: [AIRFLOW-6307] Improve 
support for a special characters in DAG's name in stats view
URL: https://github.com/apache/airflow/pull/6855#discussion_r374328889
 
 

 ##
 File path: airflow/www/views.py
 ##
 @@ -307,7 +307,7 @@ def dag_stats(self, session=None):
 
 # Filter by get parameters
 selected_dag_ids = {
-unquote(dag_id) for dag_id in request.args.get('dag_ids', 
'').split(',') if dag_id
+unquote(dag_id) for dag_id in request.args.getlist('dag_ids[]') if 
dag_id
 
 Review comment:
   I found the real specifications that describe this format.
   
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#style-examples
 last row.
   ![Screenshot 2020-02-03 at 21 39 
32](https://user-images.githubusercontent.com/12058428/73689125-ceccf500-46cd-11ea-988a-fc72bad9a2ef.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7352: [AIRFLOW-6683] Run REST API tests when DAGs are serialized

2020-02-03 Thread ASF subversion and git services (Jira)

mik-laj commented on issue #7352: [AIRFLOW-6683] Run REST API tests when DAGs 
are serialized
URL: https://github.com/apache/airflow/pull/7352#issuecomment-581605444
 
 
   Travis is sad. Can you take care of him?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4026) Add DAG Labels to improve filtering of DAGs



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029243#comment-17029243
 ] 

ASF subversion and git services commented on AIRFLOW-4026:
--

Commit 1dd857865e3e24a06a3a994c5f8db256c6a10f63 in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1dd8578 ]

fixup! fixup! [AIRFLOW-4026] Add filter by DAG tags (#6489)


> Add DAG Labels to improve filtering of DAGs
> ---
>
> Key: AIRFLOW-4026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4026
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0, 1.10.8
>
> Attachments: Screen Shot 2019-03-06 at 09.39.22.png
>
>
>  !Screen Shot 2019-03-06 at 09.39.22.png! 
> See Discussion: 
> https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551861675239000



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4026) Add DAG Labels to improve filtering of DAGs

2020-02-03 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029210#comment-17029210
 ] 

ASF subversion and git services commented on AIRFLOW-4026:
--

Commit 1e57ccb4bd03588792e190cc94d0d3a95703f6dc in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1e57ccb ]

fixup! [AIRFLOW-4026] Add filter by DAG tags (#6489)


> Add DAG Labels to improve filtering of DAGs
> ---
>
> Key: AIRFLOW-4026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4026
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0, 1.10.8
>
> Attachments: Screen Shot 2019-03-06 at 09.39.22.png
>
>
>  !Screen Shot 2019-03-06 at 09.39.22.png! 
> See Discussion: 
> https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551861675239000



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #7328: [AIRFLOW-6707] Simplify Connection.get_hook method

mik-laj commented on issue #7328: [AIRFLOW-6707] Simplify Connection.get_hook 
method
URL: https://github.com/apache/airflow/pull/7328#issuecomment-581566727
 
 
   I will gladly see your change. I would just like to point out that such a 
change should be backward compatible. Parameter `conn_id` sometimes repeat in 
operators, which also makes it difficult.  In the past, I ordered all operators 
for GCP to contain a unified parameter name - `gcp_conn_id`.
   https://issues.apache.org/jira/browse/AIRFLOW-5475
   https://issues.apache.org/jira/browse/AIRFLOW-820
   https://issues.apache.org/jira/browse/AIRFLOW-5123
   https://issues.apache.org/jira/browse/AIRFLOW-5122
   https://issues.apache.org/jira/browse/AIRFLOW-5121


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6685) Add Data Quality Operators

2020-02-03 Thread alex l (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alex l updated AIRFLOW-6685:

Description: 
Add Data Quality Operators to improve data quality testing on data 
workflows/pipelines. This includes 3 operators:
 * BaseDataQualityOperator
 ** contains shared attributes and methods that data quality check operators 
utilize
 ** a base class that can be used to create other dq operators
 * DataQualityThresholdCheckOperator
 ** will check a single value, sql result against a threshold range, and will 
fail a task if it is outside this range.
 * DataQulaityThresholdSQLCheckOperator
 ** Similar to DataQualityThresholdCheckOperator, but thresholds are 
sql-evaluated values, for dynamic threshold ranging.

  was:
Adding plugins to allow better data quality testing within a dag. This includes 
3 operators:
 * BaseDataQualityOperator
 ** contains shared attributes and methods that data quality check operators 
utilize
 ** a base class that can be used to create other dq operators
 * DataQualityThresholdCheckOperator
 ** will check a single value, sql result against a threshold range, and will 
fail a task if it is outside this range.
 * DataQulaityThresholdSQLCheckOperator
 ** Similar to DataQualityThresholdCheckOperator, but thresholds are 
sql-evaluated values, for dynamic threshold ranging.


> Add Data Quality Operators 
> ---
>
> Key: AIRFLOW-6685
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6685
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: alex l
>Assignee: alex l
>Priority: Major
>
> Add Data Quality Operators to improve data quality testing on data 
> workflows/pipelines. This includes 3 operators:
>  * BaseDataQualityOperator
>  ** contains shared attributes and methods that data quality check operators 
> utilize
>  ** a base class that can be used to create other dq operators
>  * DataQualityThresholdCheckOperator
>  ** will check a single value, sql result against a threshold range, and will 
> fail a task if it is outside this range.
>  * DataQulaityThresholdSQLCheckOperator
>  ** Similar to DataQualityThresholdCheckOperator, but thresholds are 
> sql-evaluated values, for dynamic threshold ranging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6685) Add Data Quality Operators

2020-02-03 Thread alex l (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alex l updated AIRFLOW-6685:

Summary: Add Data Quality Operators   (was: Add operators to improve data 
quality testing on data workflows/pipelines)

> Add Data Quality Operators 
> ---
>
> Key: AIRFLOW-6685
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6685
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: alex l
>Assignee: alex l
>Priority: Major
>
> Adding plugins to allow better data quality testing within a dag. This 
> includes 3 operators:
>  * BaseDataQualityOperator
>  ** contains shared attributes and methods that data quality check operators 
> utilize
>  ** a base class that can be used to create other dq operators
>  * DataQualityThresholdCheckOperator
>  ** will check a single value, sql result against a threshold range, and will 
> fail a task if it is outside this range.
>  * DataQulaityThresholdSQLCheckOperator
>  ** Similar to DataQualityThresholdCheckOperator, but thresholds are 
> sql-evaluated values, for dynamic threshold ranging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on issue #6376: [AIRFLOW-5705] Add creds backend and support for AWS SSM

ashb commented on issue #6376: [AIRFLOW-5705] Add creds backend and support for 
AWS SSM
URL: https://github.com/apache/airflow/pull/6376#issuecomment-581553006
 
 
   Ignore me and listen to Kaxil


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #6376: [AIRFLOW-5705] Add creds backend and support for AWS SSM

ashb commented on issue #6376: [AIRFLOW-5705] Add creds backend and support for 
AWS SSM
URL: https://github.com/apache/airflow/pull/6376#issuecomment-581550463
 
 
   Don't think this needs an AIP - it's not that big an architectural change. 
(This comment just from memory, haven't reviewed it recently)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6229) SparkSubmitOperator polls forever if status json can't find driverState tag



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6229:

Issue Type: Bug  (was: New Feature)

> SparkSubmitOperator polls forever if status json can't find driverState tag
> ---
>
> Key: AIRFLOW-6229
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6229
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> You click ‘release’ on a new spark cluster while the prior spark cluster is 
> processing some spark submits from airflow. Then airflow is never able to 
> finish the sparksubmit task as it polls from status on the new spark cluster 
> build which it can’t find status for as the submit happened on earlier spark 
> cluster build….the status loop goes on forever
>  
> [https://github.com/apache/airflow/blob/1.10.6/airflow/contrib/hooks/spark_submit_hook.py#L446]
> [https://github.com/apache/airflow/blob/1.10.6/airflow/contrib/hooks/spark_submit_hook.py#L489]
> It loops forever if it can’t find driverState tag in the json response, since 
> the new build (pointed to by the released DNS name) doesn’t know about the 
> driver submitted (in previously released build) then the 2nd response below 
> does not contain the driverState tag.
>   
> #response before clicking release on new build
> [ec2-user@reda ~]$
> curl +[http://dns:6066/v1/submissions/status/driver-20191202142207-]+
> {  "action" : "SubmissionStatusResponse",  "driverState" : "RUNNING",  
> "serverSparkVersion" : "2.3.4",  "submissionId" : 
> "driver-20191202142207-",  "success" : true,  "workerHostPort" : 
> "reda:31489",  "workerId" : "worker-20191202133526-reda-31489"}
>  
> #response after clicking release on new build
> [ec2-user@reda ~]$
> curl [http://dns:6066/v1/submissions/status/driver-20191202142207-]     
> {  "action" : "SubmissionStatusResponse",  "serverSparkVersion" : "2.3.4",  
> "submissionId" : "driver-20191202142207-",  "success" : false             
>   }
>                
>  
> Definitely a defect in current code. Can fix this by modifying 
> _process_spark_status_log function to set driver status to UNKNOWN if 
> driverState is not in response after iterating all lines.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (AIRFLOW-5406) Allow spark_submit_hook to run without pip install kubernetes/airflow[kubernetes]



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik reopened AIRFLOW-5406:
-

> Allow spark_submit_hook to run without pip install 
> kubernetes/airflow[kubernetes]
> -
>
> Key: AIRFLOW-5406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5406
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies, executor-kubernetes, executors
>Affects Versions: 1.10.5
>Reporter: t oo
>Assignee: t oo
>Priority: Blocker
> Fix For: 1.10.8
>
>
> Allow spark_submit_hook to run without pip install 
> kubernetes/airflow[kubernetes]
> k8s dependencies causing issues for non-k8s users based on 1.10.6:
> *settings.py:*
> from airflow.contrib.kubernetes.pod import Pod
> def pod_mutation_hook(pod):
> *models/__init__.py*
> from airflow.models.kubernetes import KubeWorkerIdentifier, 
> KubeResourceVersion
> *spark_submit_hook.py*
> from airflow.contrib.kubernetes import kube_client



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-5406) Allow spark_submit_hook to run without pip install kubernetes/airflow[kubernetes]



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-5406.
-
Resolution: Fixed

> Allow spark_submit_hook to run without pip install 
> kubernetes/airflow[kubernetes]
> -
>
> Key: AIRFLOW-5406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5406
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies, executor-kubernetes, executors
>Affects Versions: 1.10.5
>Reporter: t oo
>Assignee: t oo
>Priority: Blocker
> Fix For: 1.10.8
>
>
> Allow spark_submit_hook to run without pip install 
> kubernetes/airflow[kubernetes]
> k8s dependencies causing issues for non-k8s users based on 1.10.6:
> *settings.py:*
> from airflow.contrib.kubernetes.pod import Pod
> def pod_mutation_hook(pod):
> *models/__init__.py*
> from airflow.models.kubernetes import KubeWorkerIdentifier, 
> KubeResourceVersion
> *spark_submit_hook.py*
> from airflow.contrib.kubernetes import kube_client



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6350) security - spark submit operator logging+exceptions should mask passwords



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6350:

Issue Type: New Feature  (was: Bug)

> security - spark submit operator logging+exceptions should mask passwords 
> --
>
> Key: AIRFLOW-6350
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6350
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> contrib/hooks/spark_submit_hook.py    
> Mask passwords in spark submit cmd AND error stacktrace
>  
> *add*
> def _mask_cmd(self, connection_cmd):
>  # Mask any password related fields in application args with key value pair 
> where key contains password (case insensitive), e.g. HivePassword='abc'
> connection_cmd_masked = 
> re.sub(r"(\S*?(?:secret|password)\S*?\s*=\s*')[^']*(?=')", r'\1**', ' 
> '.join(connection_cmd), flags=re.I)
> return connection_cmd_masked
>  
> *BEFORE*
> self.log.info("Spark-Submit cmd: %s", connection_cmd)
>  
> *AFTER*
> self.log.info("Spark-Submit cmd: %s", self._mask_cmd(connection_cmd))
>  
> *BEFORE*
>  if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> spark_submit_cmd, returncode
> )
> )
>  
> *AFTER*
> if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> self._mask_cmd(spark_submit_cmd), returncode
> )
> )



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6297) Add Airflow website link to UI docs



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6297:

Fix Version/s: (was: 2.0.0)
   1.10.8

> Add Airflow website link to UI docs
> ---
>
> Key: AIRFLOW-6297
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6297
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: ui
>Affects Versions: 1.10.6
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Fix For: 1.10.8
>
> Attachments: Screen Shot 2019-12-19 at 15.44.31.png
>
>
> Add Airflow website link to UI docs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] xinbinhuang commented on issue #7328: [AIRFLOW-6707] Simplify Connection.get_hook method

xinbinhuang commented on issue #7328: [AIRFLOW-6707] Simplify 
Connection.get_hook method
URL: https://github.com/apache/airflow/pull/7328#issuecomment-581544804
 
 
   I was actually thinking about doing this the other day. On top of this, 
should we also standardize the `***_conn_id` to `conn_id` and add tests to 
enforce this in new operators? I think `conn_id` with the class operator is 
already quite self-descriptive.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] xinbinhuang edited a comment on issue #7328: [AIRFLOW-6707] Simplify Connection.get_hook method

xinbinhuang edited a comment on issue #7328: [AIRFLOW-6707] Simplify 
Connection.get_hook method
URL: https://github.com/apache/airflow/pull/7328#issuecomment-581544804
 
 
   I was actually thinking about doing this the other day. On top of this, 
would it be nice to also standardize the `***_conn_id` to `conn_id` and add 
tests to enforce this in new operators? I think `conn_id` with the class 
operator is already quite self-descriptive.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6527:

Issue Type: Improvement  (was: Bug)

> Error sending Celery task:Timeout in send_task_to_executor
> --
>
> Key: AIRFLOW-6527
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6527
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Priority: Major
> Fix For: 2.0.0, 1.10.8
>
>
> We use Airflow with CeleryExecutor and redis broker. Our airflow scheduler 
> often encounters this \{{AirflowTaskTimeout}} error. 
> - This happens in \{{send_task_to_executor()}}. 
> - It only happens occasionally. 
> - Retrying the failed task a few times always works.
> - This affects at least 1.10.6 and 1.10.7 and possibly other versions too. 
> Possible cause:
> Our airflow venv and dags_folder are on an NFS mount because we want to keep 
> the various pieces of Airflow services in sync. 
> The NFS mount can be slow sometimes. This causes the import to be slow and 
> causes \{{send_task_to_executor()}} to take more than 2 seconds.
> Other people with similar looking problems:
> The following issue is now closed. It's not clear to me whether or how the 
> user resolved this issue.
> https://github.com/bitnami/bitnami-docker-airflow-scheduler/issues/1
> Another user asked a question in the mailing list. It's not answered.
> https://www.mail-archive.com/dev@airflow.apache.org/msg01093.html
> Proposed workaround:
> - Make this `timeout(seconds=2)` configurable. E.g adding a 
> [celery]send_task_timeout to airflow.cfg. Since 2 seconds seems too short, we 
> can configure it to something like 15 seconds to make it much less likely to 
> happen.
> - Move airflow venv to the local disk. This makes it inconvenient to sync 
> airflow installation across multiple hosts though.
> {code}
> Jan 09 22:46:59 scheduler_host airflow[18882]: [2020-01-09 22:46:59,763] 
> \{celery_executor.py:224} ERROR - Error sending Celery task:Timeout, PID: 
> 27724
> Jan 09 22:46:59 scheduler_host airflow[18882]: Celery Task ID: 
> ('example_daily', 'example_sensor1', datetime.datetime(2020, 1, 9, 0, 0, 
> tzinfo=), 1)
> Jan 09 22:46:59 scheduler_host airflow[18882]: Traceback (most recent call 
> last):
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/kombu/utils/objects.py", 
> line 42, in __get__
> Jan 09 22:46:59 scheduler_host airflow[18882]: return 
> obj.__dict__[self.__name__]
> Jan 09 22:46:59 scheduler_host airflow[18882]: KeyError: 'amqp'
> Jan 09 22:46:59 scheduler_host airflow[18882]: During handling of the above 
> exception, another exception occurred:
> Jan 09 22:46:59 scheduler_host airflow[18882]: Traceback (most recent call 
> last):
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/airflow/executors/celery_executor.py",
>  line 118, in send_task_to_executor
> Jan 09 22:46:59 scheduler_host airflow[18882]: result = 
> task.apply_async(args=[command], queue=queue)
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/celery/app/task.py", line 
> 570, in apply_async
> Jan 09 22:46:59 scheduler_host airflow[18882]: **options
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/celery/app/base.py", line 
> 712, in send_task
> Jan 09 22:46:59 scheduler_host airflow[18882]: amqp = self.amqp
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/kombu/utils/objects.py", 
> line 44, in __get__
> Jan 09 22:46:59 scheduler_host airflow[18882]: value = 
> obj.__dict__[self.__name__] = self.__get(obj)
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/celery/app/base.py", line 
> 1202, in amqp
> Jan 09 22:46:59 scheduler_host airflow[18882]: return 
> instantiate(self.amqp_cls, app=self)
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/celery/utils/imports.py", 
> line 55, in instantiate
> Jan 09 22:46:59 scheduler_host airflow[18882]: return 
> symbol_by_name(name)(*args, **kwargs)
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/mnt/nfs1/airflow_venv/lib/python3.6/site-packages/kombu/utils/imports.py", 
> line 57, in symbol_by_name
> Jan 09 22:46:59 scheduler_host airflow[18882]: module = imp(module_name, 
> package=package, **kwargs)
> Jan 09 22:46:59 scheduler_host airflow[18882]: File 
> "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
> Jan 09 22:46:59 scheduler_host airflow[18882]: return 
>

[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#discussion_r374251271
 
 

 ##
 File path: tests/providers/amazon/aws/hooks/test_athena.py
 ##
 @@ -0,0 +1,120 @@
+# -*- coding: utf-8 -*-
 
 Review comment:
   That is correct.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#discussion_r374250964
 
 

 ##
 File path: airflow/providers/amazon/aws/hooks/athena.py
 ##
 @@ -133,7 +133,17 @@ def get_query_results(self, query_execution_id):
 elif query_state in self.INTERMEDIATE_STATES or query_state in 
self.FAILURE_STATES:
 self.log.error('Query is in {state} state. Cannot fetch 
results'.format(state=query_state))
 return None
-return 
self.get_conn().get_query_results(QueryExecutionId=query_execution_id)
+result_params = {'QueryExecutionId': query_execution_id,
+ 'MaxResults': max_results}
+results = self.get_conn().get_query_results(**result_params)
 
 Review comment:
   Yes that does make sense.  Now that I think about it, it probably makes 
sense to just add a new function to check that the query is successful and then 
return the paginator. For the get_query_results function maybe we should just 
adding the option to send in the NextTokenId so you can paginate manually if 
you want. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-4495:

Issue Type: New Feature  (was: Improvement)

> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> 1. 
>  useful to handle future date for externally triggered batch process where 
> ingesting 'forecast' data where filename date is in the future
> 2.
>  this error is just in the scheduler log and not propagated up, so the dag 
> stays in 'running' state forever (or for 1 year waiting for the time to pass 
> :) )
>  ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>  
>  
> fix below works if u only have externally triggered DAGs:
>  
> commenting below ti_deps\deps\runnable_exec_date_dep.py
>     #if ti.execution_date > cur_date:
>     #    yield self._failing_status(
>     #    reason="Execution date \{0} is in the future (the current "
>     #   "date is \{1}).".format(ti.execution_date.isoformat(),
>     #  cur_date.isoformat()))
>  
> commenting below jobs.py
>     # don't consider runs that are executed in the future
>     #if run.execution_date > timezone.utcnow():
>     #    self.log.error(
>     #    "Execution date is in future: %s",
>     #    run.execution_date
>     #    )
>     #    continue
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6726) Add docs on "Using airflow_local_settings.py"



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6726:

Summary: Add docs on "Using airflow_local_settings.py"  (was: Document 
using airflow_local_settings.py)

> Add docs on "Using airflow_local_settings.py"
> -
>
> Key: AIRFLOW-6726
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6726
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.9
>
>
> Document how and when to use airflow_local_settings.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil merged pull request #7350: [AIRFLOW-XXXX] Move LatestOnlyOperator change to Master

kaxil merged pull request #7350: [AIRFLOW-] Move LatestOnlyOperator change 
to Master
URL: https://github.com/apache/airflow/pull/7350
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6630) Resolve Handlebars advisory



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6630:

Fix Version/s: (was: 1.10.8)
   1.10.9

> Resolve Handlebars advisory 
> 
>
> Key: AIRFLOW-6630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6630
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.7
>Reporter: Ry Walker
>Assignee: Ry Walker
>Priority: Major
> Fix For: 1.10.9
>
>
> Security scan turned up Handlebars 4.3.5 as a vulnerability per this NPM 
> advisory [https://www.npmjs.com/advisories/1324] - I've got PR to resolve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6726) Document using airflow_local_settings.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6726:

Fix Version/s: (was: 1.10.8)

> Document using airflow_local_settings.py
> 
>
> Key: AIRFLOW-6726
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6726
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.9
>
>
> Document how and when to use airflow_local_settings.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6576:

Fix Version/s: (was: 2.0.0)

> Deleting a task with SLA crashes the scheduler
> --
>
> Key: AIRFLOW-6576
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6576
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: scheduler
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Major
> Fix For: 1.10.8
>
>
> When a task with SLA is deleted from a DAG after the SLA miss is logged but 
> before the notification was sent, scheduler will crash with an 
> AirflowException



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6576:

Issue Type: Bug  (was: New Feature)

> Deleting a task with SLA crashes the scheduler
> --
>
> Key: AIRFLOW-6576
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6576
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Major
> Fix For: 1.10.8
>
>
> When a task with SLA is deleted from a DAG after the SLA miss is logged but 
> before the notification was sent, scheduler will crash with an 
> AirflowException



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5843) Add conf form when trigger DAG from the WEB.



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5843:

Fix Version/s: (was: 2.0.0)

> Add conf form when trigger DAG from the WEB. 
> -
>
> Key: AIRFLOW-5843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5843
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: ui
>Affects Versions: 1.10.6
>Reporter: jihun.no
>Assignee: James Coder
>Priority: Minor
> Fix For: 1.10.8
>
>
> When we trigger a DAG by airflow_cli, it is possible to give conf like this.
> {code:java}
> airflow trigger_dag --conf {"file_variable": "/path/to/file"} dag_id
> {code}
>  
> But some times, Access to the webserver's shell is not easy or convenient.
> So I think, It will be very helpful if we can specify conf when trigger a DAG 
> from the airflow's web.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5843) Add conf form when trigger DAG from the WEB.