[GitHub] [airflow] aviemzur commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers

[GitBox]

aviemzur commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators 
to AWS providers
URL: https://github.com/apache/airflow/pull/6824#issuecomment-571933340
 
 
   @nuclearpinguin made the changes you requested: [Changes after CR 
2](https://github.com/apache/airflow/pull/6824/commits/e8d9972f8109a0788f708fdfbf880af88cb3fe00)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

yuqian90 commented on a change in pull request #6633: [AIRFLOW-2279] Clear 
tasks across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364092453
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -955,6 +993,60 @@ def clear(
 if only_running:
 tis = tis.filter(TI.state == State.RUNNING)
 
+if include_subdags:
+from airflow.sensors.external_task_sensor import ExternalTaskMarker
+
+# Recursively find external tasks indicated by ExternalTaskMarker
+instances = tis.all()
+for ti in instances:
+if ti.operator == ExternalTaskMarker.__name__:
+task = self.get_task(ti.task_id)
+
+if recursion_depth == 0:
+# Maximum recursion depth allowed is the 
recursion_depth of the first
+# ExternalTaskMarker in the tasks to be cleared.
+max_recursion_depth = task.recursion_depth
+
+if recursion_depth + 1 > max_recursion_depth:
+# Prevent cycles or accidents.
+raise AirflowException("Maximum recursion depth {} 
reached for {} {}. "
+   "Attempted to clear too many 
tasks "
+   "or there may be a cyclic 
dependency."
+   .format(max_recursion_depth,
+   
ExternalTaskMarker.__name__, task.task_id))
+
+instance = TaskInstance(task=task, 
execution_date=ti.execution_date)
 
 Review comment:
   You are right it's cleaner to call `ti.render_template()` (after setting 
`ti.task` to task).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS batch jobs

[GitBox]

codecov-io edited a comment on issue #6811: [AIRFLOW-6245] Add custom waiters 
for AWS batch jobs
URL: https://github.com/apache/airflow/pull/6811#issuecomment-568228799
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=h1) 
Report
   > Merging 
[#6811](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.24%`.
   > The diff coverage is `96.74%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6811/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6811  +/-   ##
   ==
   - Coverage   85.15%   84.91%   -0.25% 
   ==
 Files 680  682   +2 
 Lines   3882438938 +114 
   ==
   + Hits3306133063   +2 
   - Misses   5763 5875 +112
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...irflow/providers/amazon/aws/hooks/batch\_waiters.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF93YWl0ZXJzLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/operators/awsbatch\_operator.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hd3NiYXRjaF9vcGVyYXRvci5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/providers/amazon/aws/operators/batch.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvYmF0Y2gucHk=)
 | `93.75% <89.28%> (-2.09%)` | :arrow_down: |
   | 
[airflow/providers/amazon/aws/hooks/batch\_client.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF9jbGllbnQucHk=)
 | `96.89% <96.89%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   | ... and [1 
more](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=footer). 
Last update 
[fe20ef6...302d536](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS batch jobs

[GitBox]

codecov-io edited a comment on issue #6811: [AIRFLOW-6245] Add custom waiters 
for AWS batch jobs
URL: https://github.com/apache/airflow/pull/6811#issuecomment-568228799
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=h1) 
Report
   > Merging 
[#6811](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.24%`.
   > The diff coverage is `96.74%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6811/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6811  +/-   ##
   ==
   - Coverage   85.15%   84.91%   -0.25% 
   ==
 Files 680  682   +2 
 Lines   3882438938 +114 
   ==
   + Hits3306133063   +2 
   - Misses   5763 5875 +112
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...irflow/providers/amazon/aws/hooks/batch\_waiters.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF93YWl0ZXJzLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/contrib/operators/awsbatch\_operator.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hd3NiYXRjaF9vcGVyYXRvci5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/providers/amazon/aws/operators/batch.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvYmF0Y2gucHk=)
 | `93.75% <89.28%> (-2.09%)` | :arrow_down: |
   | 
[airflow/providers/amazon/aws/hooks/batch\_client.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF9jbGllbnQucHk=)
 | `96.89% <96.89%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   | ... and [1 
more](https://codecov.io/gh/apache/airflow/pull/6811/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=footer). 
Last update 
[fe20ef6...302d536](https://codecov.io/gh/apache/airflow/pull/6811?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use (only 2 slash):"\\\ud83c\\\udf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use (only 2 slash):"ud83cudf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps() -- 
> BaseSQLToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use (only 2 slash):"\\\ud83c\\\udf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use (only 2 slash):"ud83cudf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use (only 2 slash):"\\\ud83c\\\udf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps() -- 
> BaseSQLToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use (only 2 slash):"ud83cudf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] dazza-codes commented on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS batch jobs

[GitBox]

dazza-codes commented on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS 
batch jobs
URL: https://github.com/apache/airflow/pull/6811#issuecomment-571888722
 
 
   Did another self-review and most things look OK and test OK.  There is a 
minor revision to how the client polls the AWS Batch job description, so that 
it will fail-fast when it encounters most client errors (except one for 
too-many-requests, then it will retry).  I've briefly considered creating an 
extract-class refactor to pull out an `AwsBatchJob` class that captures only 
the job monitoring responsibility (possibly a persistent state-machine for the 
job), but skipping it because this PR already trims down the operator enough 
and that work could follow this PR if it helps later on.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dstandish edited a comment on issue #6850: [AIRFLOW-6296] add mssql odbc hook

[GitBox]

dstandish edited a comment on issue #6850: [AIRFLOW-6296] add mssql odbc hook
URL: https://github.com/apache/airflow/pull/6850#issuecomment-571734050
 
 
   also @vamega ... re your comment
   
   > I'm not sure how well known this technique is, but you can see a 
demonstration of it here
   
   incidentally, i am already using this approach in the `get_uri` method.  
note, though, this just pertains to the sqlalchemy connection string, and is 
only used in the `get_sqlalchemy_engine` method inherited from `DbApiHook`.  
   
   the `get_conn` method does not rely on sqlalchemy, but uses the same 
property `odbc_connection_string`, where the odbc string is constructed.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6975: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need)

[GitBox]

codecov-io edited a comment on issue #6975: [AIRFLOW-1467] Dynamic pooling via 
allowing tasks to use more than one pool slot (depending upon the need)
URL: https://github.com/apache/airflow/pull/6975#issuecomment-569916253
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=h1) 
Report
   > Merging 
[#6975](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/da2a617cb7494c6e358b079a6cf8c4129e742a63?src=pr=desc)
 will **increase** coverage by `0.08%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6975/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #6975  +/-   ##
   =
   + Coverage   84.81%   84.9%   +0.08% 
   =
 Files 679 680   +1 
 Lines   38491   38903 +412 
   =
   + Hits32648   33032 +384 
   - Misses   58435871  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.26% <ø> (+0.18%)` | :arrow_up: |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <ø> (ø)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `95% <100%> (+0.08%)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   | 
[airflow/utils/file.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9maWxlLnB5)
 | `86.3% <0%> (-5.17%)` | :arrow_down: |
   | ... and [97 
more](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=footer). 
Last update 
[da2a617...a51783b](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use (only 2 slash):"\\\ud83c\\\udf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:\\\ud83c\\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use (only 2 slash):"\\\ud83c\\\udf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Summary: Let emoji encoded properly for json.dumps() -- 
BaseSQLToGoogleCloudStorageOperator  (was: Let emoji encoded properly for 
json.dumps()-- BaseSQLToGoogleCloudStorageOperator)

> Let emoji encoded properly for json.dumps() -- 
> BaseSQLToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use (only 2 slash):"\\\ud83c\\\udf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:

\\ud83c and \\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:"\\ud83cudf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use:
> \\ud83c and \\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:"\\ud83cudf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:"\\ud83c\\udf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use:"\\ud83cudf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:\\\ud83c\\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:

\\ud83c and \\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use:\\\ud83c\\\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:"\\ud83c\\udf7b"

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:\\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use:"\\ud83c\\udf7b"
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:\\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c \\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use:\\ud83c\\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c \\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use: \\ud83c \\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use: \\ud83c\\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In 
[line|[https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/operators/sql_to_gcs.py#L175]]
 BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" 

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use: \\ud83c\\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83cudf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use: \\ud83cudf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damon Liao updated AIRFLOW-6505:

Description: 
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83c\\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]

  was:
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use: \\ud83cudf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]


> Let emoji encoded properly for json.dumps()-- 
> BaseSQLToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-6505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.8
> Environment: 1.10.7
>Reporter: Damon Liao
>Assignee: Damon Liao
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem 
> fixed by adding parameter `ensure_ascii=False`. 
>  
> In [line|#L175]] BaseSQLToGoogleCloudStorageOperato.
>  
> For example
> the emoji  encoded differentially when in use or not use  
> `ensure_ascii=False`.
> In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
> Not Use: \\ud83c\\\udf7b
>  
> Ref: 
> [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6505) Let emoji encoded properly for json.dumps()-- BaseSQLToGoogleCloudStorageOperator

[Damon Liao (Jira)]

Damon Liao created AIRFLOW-6505:
---

 Summary: Let emoji encoded properly for json.dumps()-- 
BaseSQLToGoogleCloudStorageOperator
 Key: AIRFLOW-6505
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6505
 Project: Apache Airflow
  Issue Type: Bug
  Components: contrib
Affects Versions: 1.10.8
 Environment: 1.10.7
Reporter: Damon Liao
Assignee: Damon Liao
 Fix For: 1.10.7


Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed 
by adding parameter `ensure_ascii=False`. 

 

In 
[line|[https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/operators/sql_to_gcs.py#L175]]
 BaseSQLToGoogleCloudStorageOperato.

 

For example

the emoji  encoded differentially when in use or not use  `ensure_ascii=False`.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" 

Not Use: \\ud83c\\udf7b

 

Ref: 
[https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook 
resolve connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571876662
 
 
   > existing tests for connection added via db/cli needs to work
   
   well, I guess the current tests don't support connection added via cli, 
right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

[GitBox]

codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once 
when running a task using StandardTaskRunner
URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) 
Report
   > Merging 
[#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `50%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7090/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7090  +/-   ##
   ==
   - Coverage   85.07%   84.79%   -0.29% 
   ==
 Files 680  680  
 Lines   3881038811   +1 
   ==
   - Hits3301932910 -109 
   - Misses   5791 5901 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `67.21% <50%> (+0.54%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). 
Last update 
[748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve 
connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571876662
 
 
   > existing tests for connection added via db/cli needs to work
   
   well, I think the current tests don't support connection added via cli, 
right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7097: [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting

[GitBox]

codecov-io edited a comment on issue #7097: [AIRFLOW-6504] Allow specifying 
configmap for Airflow Local Setting
URL: https://github.com/apache/airflow/pull/7097#issuecomment-571853779
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=h1) 
Report
   > Merging 
[#7097](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7097/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7097  +/-   ##
   ==
   + Coverage   85.15%   85.15%   +<.01% 
   ==
 Files 680  680  
 Lines   3882438832   +8 
   ==
   + Hits3306133069   +8 
 Misses   5763 5763
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5)
 | `99.29% <100%> (+0.03%)` | :arrow_up: |
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `57.57% <100%> (+0.09%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `96.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.73% <0%> (ø)` | :arrow_up: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `94.36% <0%> (ø)` | :arrow_up: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.88% <0%> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=footer). 
Last update 
[fe20ef6...a4f961b](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7097: [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting

[GitBox]

codecov-io edited a comment on issue #7097: [AIRFLOW-6504] Allow specifying 
configmap for Airflow Local Setting
URL: https://github.com/apache/airflow/pull/7097#issuecomment-571853779
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=h1) 
Report
   > Merging 
[#7097](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.06%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7097/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7097  +/-   ##
   ==
   - Coverage   85.15%   85.08%   -0.07% 
   ==
 Files 680  680  
 Lines   3882438832   +8 
   ==
   - Hits3306133042  -19 
   - Misses   5763 5790  +27
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5)
 | `99.29% <100%> (+0.03%)` | :arrow_up: |
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `57.57% <100%> (+0.09%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `95% <0%> (-1.67%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `90.08% <0%> (-1.66%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `92.95% <0%> (-1.41%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.59% <0%> (-0.29%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=footer). 
Last update 
[fe20ef6...a4f961b](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] lokeshlal commented on issue #6975: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need)

[GitBox]

lokeshlal commented on issue #6975: [AIRFLOW-1467] Dynamic pooling via allowing 
tasks to use more than one pool slot (depending upon the need)
URL: https://github.com/apache/airflow/pull/6975#issuecomment-571872342
 
 
   Hi @dimberman, I have done changes in few more test cases to pass 
pool_capacity property in the Mock object. Test cases also executed 
successfully. Could you please review the latest commit as well (commit after 
your approval).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

[Qian Yu (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010303#comment-17010303
 ] 

Qian Yu commented on AIRFLOW-6497:
--

[~kamil.bregula] Do you want to create a PR? Or if you can give more details on 
how it should be done, I can give it a try too.

> Scheduler creates DagBag in the same process with outdated info
> ---
>
> Key: AIRFLOW-6497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6497
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Priority: Major
>
> The following code in scheduler_job.py seems to be called in the same process 
> as the scheduler. It creates a DagBag. But since scheduler is a long running 
> process, it does not pick up the latest changes made to DAGs. For example, 
> changes to retries count, on_failure_callback, newly added tasks, etc are not 
> reflected.
>  
> {code:python}
> if ti.try_number == try_number and ti.state == State.QUEUED:
> msg = ("Executor reports task instance {} finished ({}) "
>"although the task says its {}. Was the task "
>"killed externally?".format(ti, state, ti.state))
> Stats.incr('scheduler.tasks.killed_externally')
> self.log.error(msg)
> try:
> simple_dag = simple_dag_bag.get_dag(dag_id)
> dagbag = models.DagBag(simple_dag.full_filepath)
> dag = dagbag.get_dag(dag_id)
> ti.task = dag.get_task(task_id)
> ti.handle_failure(msg)
> except Exception:
> self.log.error("Cannot load the dag bag to handle 
> failure for %s"
>". Setting task to FAILED without 
> callbacks or "
>"retries. Do you have enough 
> resources?", ti)
> ti.state = State.FAILED
> session.merge(ti)
> session.commit()
> {code}
> This causes errors such as AttributeError due to stale code being hit. E.g. 
> when someone added a .join attribute to CustomOperator without bouncing the 
> scheduler, this is what he would get after a CeleryWorker timeout error 
> causes this line to be hit:
> {code}
> [2020-01-05 22:25:45,951] {dagbag.py:207} ERROR - Failed to import: 
> /dags/dag1.py
> Traceback (most recent call last):
>   File "/lib/python3.6/site-packages/airflow/models/dagbag.py", line 204, in 
> process_file
> m = imp.load_source(mod_name, filepath)
>   File "/usr/lib/python3.6/imp.py", line 172, in load_source
> module = _load(spec)
>   File "", line 684, in _load
>   File "", line 665, in _load_unlocked
>   File "", line 678, in exec_module
>   File "", line 219, in _call_with_frames_removed
>   File "/dags/dag1.py", line 280, in 
> task1 >> task2.join
> AttributeError: 'CustomOperator' object has no attribute 'join'
> [2020-01-05 22:25:45,951] {scheduler_job.py:1314} ERROR - Cannot load the dag 
> bag to handle failure for  [queued]>. Setting task to FAILED without callbacks or retries. Do you have 
> enough resou
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks 
across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364042758
 
 

 ##
 File path: airflow/example_dags/example_external_task_marker.py
 ##
 @@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example DAG demonstrating setting up inter-DAG dependencies using 
ExternalTaskSensor and
+ExternalTaskMarker
+
+In this example, child_task1 in example_external_task_marker_child depends on 
parent_task in
+example_external_task_marker_parent. When parent_task is cleared with 
"Recursive" selected,
+the presence of ExternalTaskMarker tells Airflow to clear child_task1 and its
+downstream tasks.
+"""
+
+import pendulum
+
+from airflow.models import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.sensors.external_task_sensor import ExternalTaskMarker, 
ExternalTaskSensor
+
+start_date = pendulum.parse("20150101")
 
 Review comment:
   let's use the time object instead of parsing it from string


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks 
across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364043684
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -955,6 +993,60 @@ def clear(
 if only_running:
 tis = tis.filter(TI.state == State.RUNNING)
 
+if include_subdags:
+from airflow.sensors.external_task_sensor import ExternalTaskMarker
+
+# Recursively find external tasks indicated by ExternalTaskMarker
+instances = tis.all()
+for ti in instances:
+if ti.operator == ExternalTaskMarker.__name__:
+task = self.get_task(ti.task_id)
+
+if recursion_depth == 0:
+# Maximum recursion depth allowed is the 
recursion_depth of the first
+# ExternalTaskMarker in the tasks to be cleared.
+max_recursion_depth = task.recursion_depth
+
+if recursion_depth + 1 > max_recursion_depth:
+# Prevent cycles or accidents.
+raise AirflowException("Maximum recursion depth {} 
reached for {} {}. "
+   "Attempted to clear too many 
tasks "
+   "or there may be a cyclic 
dependency."
+   .format(max_recursion_depth,
+   
ExternalTaskMarker.__name__, task.task_id))
+
+instance = TaskInstance(task=task, 
execution_date=ti.execution_date)
 
 Review comment:
   Where is `instance` used?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks 
across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364043507
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -955,6 +993,60 @@ def clear(
 if only_running:
 tis = tis.filter(TI.state == State.RUNNING)
 
+if include_subdags:
+from airflow.sensors.external_task_sensor import ExternalTaskMarker
+
+# Recursively find external tasks indicated by ExternalTaskMarker
+instances = tis.all()
+for ti in instances:
+if ti.operator == ExternalTaskMarker.__name__:
+task = self.get_task(ti.task_id)
+
+if recursion_depth == 0:
+# Maximum recursion depth allowed is the 
recursion_depth of the first
+# ExternalTaskMarker in the tasks to be cleared.
+max_recursion_depth = task.recursion_depth
+
+if recursion_depth + 1 > max_recursion_depth:
+# Prevent cycles or accidents.
+raise AirflowException("Maximum recursion depth {} 
reached for {} {}. "
+   "Attempted to clear too many 
tasks "
+   "or there may be a cyclic 
dependency."
+   .format(max_recursion_depth,
+   
ExternalTaskMarker.__name__, task.task_id))
+
+instance = TaskInstance(task=task, 
execution_date=ti.execution_date)
 
 Review comment:
   Can we not directly do `ti.render_templates` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks 
across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364042612
 
 

 ##
 File path: airflow/example_dags/example_external_task_marker.py
 ##
 @@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
 
 Review comment:
   Although it is redundant but that is the convention of other dags too


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

kaxil commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks 
across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r364042552
 
 

 ##
 File path: airflow/example_dags/example_external_task_marker.py
 ##
 @@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
 
 Review comment:
   Let's change filename to 
`airflow/example_dags/example_external_task_marker_dag.py`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

codecov-io edited a comment on issue #7098: [AIRFLOW-4453] Make behavior of 
`none_failed` consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571862145
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=h1) 
Report
   > Merging 
[#7098](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.27%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7098/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7098  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.28% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132951 -110 
   - Misses   5763 5871 +108
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==)
 | `93.24% <ø> (+2.45%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=footer). 
Last update 
[fe20ef6...19f9fe8](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

codecov-io commented on issue #7098: [AIRFLOW-4453] Make behavior of 
`none_failed` consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571862145
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=h1) 
Report
   > Merging 
[#7098](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.27%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7098/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7098  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.28% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132951 -110 
   - Misses   5763 5871 +108
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==)
 | `93.24% <ø> (+2.45%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7098/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=footer). 
Last update 
[fe20ef6...19f9fe8](https://codecov.io/gh/apache/airflow/pull/7098?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` 
consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571861953
 
 
   Adding a note. I'm trying to think of a workaround if anyone actually was 
relying on the previous behavior. I don't know if anyone was, but if they were, 
we may want to add another `trigger_rule` that explicitly implements that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

[Jarek Potiuk (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010259#comment-17010259
 ] 

Jarek Potiuk commented on AIRFLOW-2516:
---

Yep. They are the same. That 's what you would expect.

> Deadlock found when trying to update task_instance table
> 
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
>Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 
> 1.10.5, 1.10.6, 1.10.7
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png, 
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, 
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py, 
> jobs_fixed_deadlock_possibly_1.9.py, 
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>  
>  
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: 
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: 
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227 
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback 
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/bin/airflow", line 27, in 
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in 
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = 
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in 
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in 
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: 
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = 
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, 
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, 
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, 
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, 
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, 
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, 
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, 
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return 
> self._execute_and_instances(context)
> [2018-05-23 

[jira] [Updated] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Joel Croteau (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Croteau updated AIRFLOW-4453:
--
Attachment: 3_step.png

> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: 3_step.png, cube_update.py, 
> image-2019-05-02-18-11-28-307.png, simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Joel Croteau (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010257#comment-17010257
 ] 

Joel Croteau commented on AIRFLOW-4453:
---

[~kamil.bregula], added an extra step with a dummy rule to my test workflow. 
`step_3` executed. The rest of the behavior was unchanged.

 

!3_step.png!

> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: 3_step.png, cube_update.py, 
> image-2019-05-02-18-11-28-307.png, simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

mik-laj commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` 
consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571854915
 
 
   I think that a note in UPDATING.md could also be useful.  Some people may 
have DAGs that relied on this condition, so we should tell them how to update 
their DAGs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` 
consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571854759
 
 
   Yes, but not right now. Might be a little while before I have time.
   
   On Tue, Jan 7, 2020, 5:45 PM Kamil Breguła  wrote:
   
   > Can you add tests to avoid regression?
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

mik-laj commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` 
consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-571854505
 
 
   Can you add tests to avoid regression?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7097: [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting

[GitBox]

codecov-io commented on issue #7097: [AIRFLOW-6504] Allow specifying configmap 
for Airflow Local Setting
URL: https://github.com/apache/airflow/pull/7097#issuecomment-571853779
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=h1) 
Report
   > Merging 
[#7097](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7097/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7097  +/-   ##
   ==
   + Coverage   85.15%   85.15%   +<.01% 
   ==
 Files 680  680  
 Lines   3882438832   +8 
   ==
   + Hits3306133069   +8 
 Misses   5763 5763
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5)
 | `99.29% <100%> (+0.03%)` | :arrow_up: |
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7097/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `57.57% <100%> (+0.09%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=footer). 
Last update 
[fe20ef6...962ee8a](https://codecov.io/gh/apache/airflow/pull/7097?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010244#comment-17010244
 ] 

ASF GitHub Bot commented on AIRFLOW-4453:
-

TV4Fun commented on pull request #7098: [AIRFLOW-4453] Make behavior of 
`none_failed` consistent with documentation
URL: https://github.com/apache/airflow/pull/7098
 
 
   The documentation for the `none_failed` trigger rule
   (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules)
   describes its behavior as "all parents have not failed (`failed` or
   `upstream_failed`) i.e. all parents have succeeded or been skipped."
   With that definition in mind, there is no reason that the check for
   `none_failed` should ever have to check for skipped upstream tasks or
   set the current task to `skipped`. The current behavior causes the rule
   to skip if all upstream tasks have skipped, which is more than a little
   confusing. This fixes the behavior to be consistent with the documentation.
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [X] Description above provides context of the change
   - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] TV4Fun opened a new pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation

[GitBox]

TV4Fun opened a new pull request #7098: [AIRFLOW-4453] Make behavior of 
`none_failed` consistent with documentation
URL: https://github.com/apache/airflow/pull/7098
 
 
   The documentation for the `none_failed` trigger rule
   (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules)
   describes its behavior as "all parents have not failed (`failed` or
   `upstream_failed`) i.e. all parents have succeeded or been skipped."
   With that definition in mind, there is no reason that the check for
   `none_failed` should ever have to check for skipped upstream tasks or
   set the current task to `skipped`. The current behavior causes the rule
   to skip if all upstream tasks have skipped, which is more than a little
   confusing. This fixes the behavior to be consistent with the documentation.
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [X] Description above provides context of the change
   - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run

[GitBox]

mik-laj commented on issue #6993: [AIRFLOW-4502] new cli command - 
task_states_for_dag_run
URL: https://github.com/apache/airflow/pull/6993#issuecomment-571853166
 
 
   Each test should delete and create data in the database, but unfortunately, 
we have one database that is shared by all tests, so if one test adds data to 
the database, another can read them. Side effects may occur between tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Kamil Bregula (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010240#comment-17010240
 ] 

Kamil Bregula commented on AIRFLOW-4453:


Perfectly. Can you create PR then? 

> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Kamil Bregula (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010239#comment-17010239
 ] 

Kamil Bregula commented on AIRFLOW-4453:


Can you check what happens if you add one more task at the end?

{code:python}
step_3 = DummyOperator(task_id='step_3, trigger_rule='dummy')
step 1 >> step_3
{code}


> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

[Zhidong (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010233#comment-17010233
 ] 

Zhidong commented on AIRFLOW-2516:
--

The execution plans for *SELECT ... WHERE  FOR UPDATE* and *UPDATE 
... WHERE * are identical.
{code:java}
EXPLAIN UPDATE task_instance, dag_run SET task_instance.state=NULL WHERE 
task_instance.dag_id IN ('test_dag_1', 'test_dag_2') AND task_instance.state IN 
('queued', 'scheduled') AND dag_run.dag_id = task_instance.dag_id AND 
dag_run.execution_date = task_instance.execution_date AND dag_run.state != 
'running';+--+-+---+--+++-+---+--+-+|
 id   | select_type | table         | type | possible_keys                      
| key    | key_len | ref                                                        
       | rows | Extra       
|+--+-+---+--+++-+---+--+-+|
    1 | SIMPLE      | task_instance | ALL  | ti_dag_state,ti_state_lkp,ti_state 
| NULL   | NULL    | NULL                                                       
       | 1    | Using where ||    1 | SIMPLE      | dag_run       | ref  | 
dag_id,dag_id_2,dag_id_state       | dag_id | 262     | 
airflow.task_instance.dag_id,airflow.task_instance.execution_date | 1    | 
Using where 
|+--+-+---+--+++-+---+--+-+
EXPLAIN SELECT * FROM task_instance, dag_run WHERE task_instance.dag_id IN 
('test_dag_1', 'test_dag_2') AND task_instance.state IN ('queued', 'scheduled') 
AND dag_run.dag_id = task_instance.dag_id AND dag_run.execution_date = 
task_instance.execution_date AND dag_run.state != 'running' FOR 
UPDATE;+--+-+---+--+++-+---+--+-+|
 id   | select_type | table         | type | possible_keys                      
| key    | key_len | ref                                                        
       | rows | Extra       
|+--+-+---+--+++-+---+--+-+|
    1 | SIMPLE      | task_instance | ALL  | ti_dag_state,ti_state_lkp,ti_state 
| NULL   | NULL    | NULL                                                       
       | 1    | Using where ||    1 | SIMPLE      | dag_run       | ref  | 
dag_id,dag_id_2,dag_id_state       | dag_id | 262     | 
airflow.task_instance.dag_id,airflow.task_instance.execution_date | 1    | 
Using where 
|+--+-+---+--+++-+---+--+-+
{code}
I think that's why mysql document says they are same regarding locks.

> Deadlock found when trying to update task_instance table
> 
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
>Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 
> 1.10.5, 1.10.6, 1.10.7
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png, 
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, 
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py, 
> jobs_fixed_deadlock_possibly_1.9.py, 
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>  
>  
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: 
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: 
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227 
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback 
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/bin/airflow", line 27, in 
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> pool=args.pool,

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Joel Croteau (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010232#comment-17010232
 ] 

Joel Croteau commented on AIRFLOW-4453:
---

The problem seems to be [this 
code|https://github.com/kaxil/airflow/blob/master/airflow/ti_deps/deps/trigger_rule_dep.py#L157]:

 
{code:python}
elif tr == TR.NONE_FAILED:
if upstream_failed or failed:
ti.set_state(State.UPSTREAM_FAILED, session)
elif skipped == upstream:
ti.set_state(State.SKIPPED, session)
{code}

This causes a task to skip if all upstream dependencies are skipped, which 
directly contradicts [the 
documentation|https://airflow.apache.org/docs/stable/concepts.html#trigger-rules],
 which says "`none_failed`: all parents have not failed (`failed` or 
`upstream_failed`) i.e. all parents have succeeded or been skipped." There 
shouldn't be any reason that `none_failed` should ever have to look at skipped 
upstream tasks, so we should really just get rid of the second check altogether.

> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Kamil Bregula (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010228#comment-17010228
 ] 

Kamil Bregula commented on AIRFLOW-4453:


I am so sorry, my bad. I will correct it in a moment then.


> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Kamil Bregula (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula reopened AIRFLOW-4453:


> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io commented on issue #7096: [AIRFLOW-6451] self._print_stat() in 
dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

codecov-io edited a comment on issue #7096: [AIRFLOW-6451] self._print_stat() 
in dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096#issuecomment-571842522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=h1) 
Report
   > Merging 
[#7096](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7096/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7096  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.29% 
   ==
 Files 680  680  
 Lines   3882438822   -2 
   ==
   - Hits3306132949 -112 
   - Misses   5763 5873 +110
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.95% <100%> (-0.05%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7096/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=footer). 
Last update 
[fe20ef6...e6c6ce8](https://codecov.io/gh/apache/airflow/pull/7096?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5704) Docker scripts for kind kubernetes tests can be improved

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010218#comment-17010218
 ] 

ASF GitHub Bot commented on AIRFLOW-5704:
-

potiuk commented on pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes 
scripts for local testing
URL: https://github.com/apache/airflow/pull/6516
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Docker scripts for kind kubernetes tests can be improved
> 
>
> Key: AIRFLOW-5704
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5704
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>
> The docker CI image for kind tests can be improved
>  
>  * Kubernetes Version and all the installation of docker + kubectl + kind can 
> be added back
>  * Running kubernetes scripts should be possible from within breeze without 
> special "kubernetes" environment
>  * --env breeze switch should be removed
>  * "bare" environment should be replaced by --no-deps switch
>  * ENV variable should disappear
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5704) Docker scripts for kind kubernetes tests can be improved

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010219#comment-17010219
 ] 

ASF GitHub Bot commented on AIRFLOW-5704:
-

potiuk commented on pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes 
scripts for local testing
URL: https://github.com/apache/airflow/pull/6516
 
 
   Note only last commit is important as this depends on #7084 
   
   * Fixed problem that Kubernetes tests were testing latest master
 rather than what came from the local sources.
   * Moved Kubernetes scripts to 'in_container' dir where they belong now
   * Kubernetes tests are now better suited for running locally
   * Kubernetes cluster is not deleted until environment is stopped
   * Kubernetes image is built outside of the container and passed as .tar
   * Kubectl version name is corrected in the Dockerfile
   * Kubernetes Version can be used to select Kubernetes versio
   * Running kubernetes scripts is now easy in Breeze
   * Instructions on how to run Kubernetes tests are updated
   * Better flags in Breeze are used to run Kubernetes environment/tests
   * The old "bare" environment is replaced by --no-deps switch
   
   ---
   Issue link: 
[AIRFLOW-5704](https://issues.apache.org/jira/browse/AIRFLOW-5704/)
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Docker scripts for kind kubernetes tests can be improved
> 
>
> Key: AIRFLOW-5704
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5704
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>
> The docker CI image for kind tests can be improved
>  
>  * Kubernetes Version and all the installation of docker + kubectl + kind can 
> be added back
>  * Running kubernetes scripts should be possible from within breeze without 
> special "kubernetes" environment
>  * --env breeze switch should be removed
>  * "bare" environment should be replaced by --no-deps switch
>  * ENV variable should disappear
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk closed pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing

[GitBox]

potiuk closed pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes 
scripts for local testing
URL: https://github.com/apache/airflow/pull/6516
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk opened a new pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing

[GitBox]

potiuk opened a new pull request #6516: [AIRFLOW-5704] Improve Kind Kubernetes 
scripts for local testing
URL: https://github.com/apache/airflow/pull/6516
 
 
   Note only last commit is important as this depends on #7084 
   
   * Fixed problem that Kubernetes tests were testing latest master
 rather than what came from the local sources.
   * Moved Kubernetes scripts to 'in_container' dir where they belong now
   * Kubernetes tests are now better suited for running locally
   * Kubernetes cluster is not deleted until environment is stopped
   * Kubernetes image is built outside of the container and passed as .tar
   * Kubectl version name is corrected in the Dockerfile
   * Kubernetes Version can be used to select Kubernetes versio
   * Running kubernetes scripts is now easy in Breeze
   * Instructions on how to run Kubernetes tests are updated
   * Better flags in Breeze are used to run Kubernetes environment/tests
   * The old "bare" environment is replaced by --no-deps switch
   
   ---
   Issue link: 
[AIRFLOW-5704](https://issues.apache.org/jira/browse/AIRFLOW-5704/)
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6504) KubernetesExecutor: Allow specifying configmap for Airflow Local Setting

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010212#comment-17010212
 ] 

ASF GitHub Bot commented on AIRFLOW-6504:
-

kaxil commented on pull request #7097: [AIRFLOW-6504] Allow specifying 
configmap for Airflow Local Setting
URL: https://github.com/apache/airflow/pull/7097
 
 
   Currently `airflow.cfg` file can be passed via ConfigMap using 
`airflow_configmap` under `[kubernetes]` section setting.
   
   We want to be able to specify `airflow_local_settings.py` via configmap too 
so that the Kubernetes Worker has this file. We need 
`airflow_local_settings.py` to specify pod_mutation_hook. 
   
   Without that currently, if we are using `KubernetesExecutor` and have a task 
with `KubernetesPodOperator`, it will inherit and change Pod using 
`pod_mutation_hook`
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> KubernetesExecutor: Allow specifying configmap for Airflow Local Setting
> 
>
> Key: AIRFLOW-6504
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6504
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.8
>
>
> Currently airflow.cfg file can be passed via ConfigMap to `airflow_configmap` 
> setting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7097: [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting

[GitBox]

kaxil opened a new pull request #7097: [AIRFLOW-6504] Allow specifying 
configmap for Airflow Local Setting
URL: https://github.com/apache/airflow/pull/7097
 
 
   Currently `airflow.cfg` file can be passed via ConfigMap using 
`airflow_configmap` under `[kubernetes]` section setting.
   
   We want to be able to specify `airflow_local_settings.py` via configmap too 
so that the Kubernetes Worker has this file. We need 
`airflow_local_settings.py` to specify pod_mutation_hook. 
   
   Without that currently, if we are using `KubernetesExecutor` and have a task 
with `KubernetesPodOperator`, it will inherit and change Pod using 
`pod_mutation_hook`
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks

[Joel Croteau (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010208#comment-17010208
 ] 

Joel Croteau commented on AIRFLOW-4453:
---

[~kamil.bregula], this ticket is about a bug in the `none_failed` trigger rule. 
The issue you linked is about `none_skipped`. I'm not seeing anything in your 
PR that changes the behavior of `none_failed`. Can you explained what was fixed 
here?

> none_failed trigger rule cascading skipped state to downstream tasks
> 
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
>  Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, 
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to 
> downstream task
>  * task have multiple upstream tasks
>  * trigger_rule set to 'none_failed'
>  * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
>  !image-2019-05-02-18-11-28-307.png|width=655,height=372! 
>  DAG attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

[t oo (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-6451 started by t oo.
-
> self._print_stat() in dag_processing.py should be skippable by config option
> 
>
> Key: AIRFLOW-6451
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6451
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
>
> perf benefit
> clean up extra poll, logs, typos



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tooptoop4 commented on issue #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run

[GitBox]

tooptoop4 commented on issue #6993: [AIRFLOW-4502] new cli command - 
task_states_for_dag_run
URL: https://github.com/apache/airflow/pull/6993#issuecomment-571833089
 
 
   @mik-laj do u think tests/www/test_views.py test is failing due to commands 
being run in tests/cli/task_command.py? I thought each test is 'self-contained'


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010188#comment-17010188
 ] 

ASF GitHub Bot commented on AIRFLOW-6451:
-

tooptoop4 commented on pull request #7096: [AIRFLOW-6451] self._print_stat() in 
dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> self._print_stat() in dag_processing.py should be skippable by config option
> 
>
> Key: AIRFLOW-6451
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6451
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Minor
>
> perf benefit
> clean up extra poll, logs, typos



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tooptoop4 opened a new pull request #7096: [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable

[GitBox]

tooptoop4 opened a new pull request #7096: [AIRFLOW-6451] self._print_stat() in 
dag_processing.py should be skippable
URL: https://github.com/apache/airflow/pull/7096
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6504) KubernetesExecutor: Allow specifying configmap for Airflow Local Setting

[Kaxil Naik (Jira)]

Kaxil Naik created AIRFLOW-6504:
---

 Summary: KubernetesExecutor: Allow specifying configmap for 
Airflow Local Setting
 Key: AIRFLOW-6504
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6504
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 1.10.7
Reporter: Kaxil Naik
Assignee: Kaxil Naik
 Fix For: 1.10.8


Currently airflow.cfg file can be passed via ConfigMap to `airflow_configmap` 
setting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6322) Add pytest marker for slow tests

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010165#comment-17010165
 ] 

ASF GitHub Bot commented on AIRFLOW-6322:
-

dazza-codes commented on pull request #6876: [AIRFLOW-6322] Add 
pytest.mark.slow marker
URL: https://github.com/apache/airflow/pull/6876
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add pytest marker for slow tests
> 
>
> Key: AIRFLOW-6322
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6322
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 1.10.6
>Reporter: Darren Weber
>Assignee: Darren Weber
>Priority: Minor
> Fix For: 1.10.8
>
>
> Add pytest marker for slow tests; see also broader context of markers 
> discussed at
> - 
> [https://lists.apache.org/thread.html/4538437c96f599766005ba7829d0bee1511debb4f53599e0d300a56f%40%3Cdev.airflow.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] dazza-codes closed pull request #6876: [AIRFLOW-6322] Add pytest.mark.slow marker

[GitBox]

dazza-codes closed pull request #6876: [AIRFLOW-6322] Add pytest.mark.slow 
marker
URL: https://github.com/apache/airflow/pull/6876
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dazza-codes commented on issue #6876: [AIRFLOW-6322] Add pytest.mark.slow marker

[GitBox]

dazza-codes commented on issue #6876: [AIRFLOW-6322] Add pytest.mark.slow marker
URL: https://github.com/apache/airflow/pull/6876#issuecomment-571824125
 
 
   The marker definitions will be decided and defined outside of this PR - 
closing this in deference to something else.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6503) support more schedule expressions (ie last day of the month)

[t oo (Jira)]

t oo created AIRFLOW-6503:
-

 Summary: support more schedule expressions (ie last day of the 
month)
 Key: AIRFLOW-6503
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6503
 Project: Apache Airflow
  Issue Type: New Feature
  Components: scheduler
Affects Versions: 1.10.7
Reporter: t oo


 ability to specify schedule like 11am on '2nd weekday of the month', 'last 5 
days of the month', 'last business day of the month'

one example:
https://confluence.atlassian.com/servicedeskserver030/constructing-cron-expressions-for-a-filter-subscription-761769111.html?_ga=2.19102328.1975309919.1578439629-428410701.1578439629



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tooptoop4 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run

[GitBox]

tooptoop4 commented on a change in pull request #4751: [AIRFLOW-3607] collected 
trigger rule dep check per dag run
URL: https://github.com/apache/airflow/pull/4751#discussion_r363999098
 
 

 ##
 File path: airflow/models/dagrun.py
 ##
 @@ -263,48 +262,34 @@ def update_state(self, session=None):
 Determines the overall state of the DagRun based on the state
 of its TaskInstances.
 
+:param finished_tasks: The finished tasks of this run
+:type finished_tasks: list[airflow.models.TaskInstance]
 :return: State
 """
 
 dag = self.get_dag()
-
-tis = self.get_task_instances(session=session)
-self.log.debug("Updating state for %s considering %s task(s)", self, 
len(tis))
+ready_tis = []
+tis = [ti for ti in self.get_task_instances(session=session) if 
ti.state != State.REMOVED]
 
 Review comment:
   do you have timings on performance benefit in a larger DAG say 500 tasks or 
5000 tasks?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator

[GitBox]

codecov-io edited a comment on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] 
Implement LatestOnlyOperator as BaseBranchOperator
URL: https://github.com/apache/airflow/pull/5970#issuecomment-571816542
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=h1) 
Report
   > Merging 
[#5970](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.27%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5970/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5970  +/-   ##
   ==
   - Coverage   85.15%   84.87%   -0.28% 
   ==
 Files 680  680  
 Lines   3882438823   -1 
   ==
   - Hits3306132952 -109 
   - Misses   5763 5871 +108
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=)
 | `100% <100%> (+10%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `88.43% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=footer). 
Last update 
[fe20ef6...ceaf360](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run

[GitBox]

tooptoop4 commented on a change in pull request #4751: [AIRFLOW-3607] collected 
trigger rule dep check per dag run
URL: https://github.com/apache/airflow/pull/4751#discussion_r363998318
 
 

 ##
 File path: airflow/models/dagrun.py
 ##
 @@ -263,48 +262,34 @@ def update_state(self, session=None):
 Determines the overall state of the DagRun based on the state
 of its TaskInstances.
 
+:param finished_tasks: The finished tasks of this run
+:type finished_tasks: list[airflow.models.TaskInstance]
 :return: State
 """
 
 dag = self.get_dag()
-
-tis = self.get_task_instances(session=session)
-self.log.debug("Updating state for %s considering %s task(s)", self, 
len(tis))
+ready_tis = []
+tis = [ti for ti in self.get_task_instances(session=session) if 
ti.state != State.REMOVED]
 
 Review comment:
   did u notice REMOVED task used to count towards SUCCESS dagrun? I raised 
https://github.com/apache/airflow/pull/6954


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6502) Support swarm networks in DockerSwarmOperator

[Arseniy (Jira)]

Arseniy created AIRFLOW-6502:


 Summary: Support swarm networks in DockerSwarmOperator
 Key: AIRFLOW-6502
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6502
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib, operators
Affects Versions: 1.10.7
 Environment: Reproducible everywhere since missing feature
Reporter: Arseniy
Assignee: Arseniy


DockerSwarmOperator does not support attaching the created ephemeral services 
to existing networks. This is supported in the Docker python API so this should 
be an easy PR. I'll do it, but contrib guidelines suggest making a Jira ticket 
first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator

[GitBox]

codecov-io commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement 
LatestOnlyOperator as BaseBranchOperator
URL: https://github.com/apache/airflow/pull/5970#issuecomment-571816542
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=h1) 
Report
   > Merging 
[#5970](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fe20ef6147f157b7c963ad46e13aff7c16458632?src=pr=desc)
 will **decrease** coverage by `0.85%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5970/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #5970  +/-   ##
   =
   - Coverage   85.15%   84.3%   -0.86% 
   =
 Files 680 680  
 Lines   38824   38823   -1 
   =
   - Hits33061   32728 -333 
   - Misses   57636095 +332
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=)
 | `100% <100%> (+10%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `49.65% <0%> (-38.78%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=footer). 
Last update 
[fe20ef6...ceaf360](https://codecov.io/gh/apache/airflow/pull/5970?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6501) UI - show next_execution date, first exec_date, creation date (ie timestamp when first parsed into dagbag)

[t oo (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

t oo updated AIRFLOW-6501:
--
Summary: UI - show next_execution date, first exec_date, creation date (ie 
timestamp when first parsed into dagbag)  (was: UI - show next_execution date)

> UI - show next_execution date, first exec_date, creation date (ie timestamp 
> when first parsed into dagbag)
> --
>
> Key: AIRFLOW-6501
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6501
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: ui
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Minor
>
> CLI has next_execution command
> would be great if UI had a page to show next execution date per dag



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] dazza-codes commented on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS batch jobs

[GitBox]

dazza-codes commented on issue #6811: [AIRFLOW-6245] Add custom waiters for AWS 
batch jobs
URL: https://github.com/apache/airflow/pull/6811#issuecomment-571814074
 
 
   Rebased and pushed again, travis failed one test with a timeout or 
something, not something specific to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6501) UI - show next_execution date

[t oo (Jira)]

t oo created AIRFLOW-6501:
-

 Summary: UI - show next_execution date
 Key: AIRFLOW-6501
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6501
 Project: Apache Airflow
  Issue Type: New Feature
  Components: ui
Affects Versions: 1.10.7
Reporter: t oo


CLI has next_execution command
would be great if UI had a page to show next execution date per dag



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-1510) Scheduler: priority_weight sorting not applied across multiple DAGs

[Luke Bodeen (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010141#comment-17010141
 ] 

Luke Bodeen commented on AIRFLOW-1510:
--

another idea like [~JosephHarris] said was maybe we remove the limitation of 
looking at only max_threads number of DAGS for consideration in scheduling 
tasks.. 

 

That could have other implications around performance, memory etc...

> Scheduler: priority_weight sorting not applied across multiple DAGs
> ---
>
> Key: AIRFLOW-1510
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1510
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.8.1
> Environment: Ubuntu, CeleryRunner
>Reporter: Joseph Harris
>Priority: Major
>
> h3. Issue
> When there are multiple available tasks across many DAGs, the order in which 
> those tasks are queued does not respect the priority_weighting order across 
> all available tasks. The run order is instead dependent on how quickly the 
> Scheduler loop reaches the DAG, when the number of DAGs is greater than the 
> scheduler 'max_threads' variable.
> This is particularly problematic when there are long-running tasks competing 
> over a limited slots in a pool.
> With over 80 DAGs in operations, increasing max_threads to this number 
> doesn't seem like a practical solution.
> h3. What should be done
> * The docs should be updated to be less misleading about how priority_weight 
> is likely to behave: https://airflow.incubator.apache.org/concepts.html#pools
> * Potential implementation improvements on the scheduler: force the 
> ProcessorManager to wait for all jobs to be processed (slow but reliable) - 
> or make _execute_task_instances() look at tasks from all DAGs (faster but 
> less reliable).
> h3. Example
> For instance, with 4 tasks:
> || DAG ||Task||Priority||Pool||
> || A || 1 |20|pool|
> || B || 2 |1|pool|
> || C || 3 |1|pool|
> || D || 4 |100|pool|
> The scheduler would look at DAGs A & B first, and send the tasks in order (1, 
> 2). Then the scheduler would look at 3 & 4 and send these in order (4, 3) if 
> there are enough pool slots available.
> h3. Current Implementation Detail
> The SchedulerJob code is a bit complex, but the sequence of events in the 
> scheduler loop looks like this:
> * The 
> [DagFileProcessorManager|https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/utils/dag_processing.py#L298]
>  loops across all discovered DAG files, and launches Processor threads 
> (limited by max_threads). Each processor reads in a single DAG, and checks 
> whether any tasks in the DAG have the dependencies met. If dependencies are 
> met, the task is set to state='scheduled'.
> * 
> [DagFileProcessorManager.heartbeat()|https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/jobs.py#L1409]
>  returns a list of DAGs returned by the Processor threads during its last 
> cycle. When max_threads = 2, this list will contain a maximum of 2 DAGs.
> * These DAGs are [passed to 
> SchedulerJob._execute_task_instances()|https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/jobs.py#L1440]
> * An [ORM 
> query|https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/jobs.py#L992]
>  selects tasks where state='scheduled' and the task is in the DAGs returned 
> by the last heartbeat() loop. *Only those tasks are [sorted and passed to the 
> Celery 
> queue|https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/jobs.py#L1035]*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6500) dependency 'Execution Date' FAILED: Execution date is in the future

[Pavan Maguluri (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavan Maguluri updated AIRFLOW-6500:

Priority: Major  (was: Minor)

> dependency 'Execution Date' FAILED: Execution date is in the future
> ---
>
> Key: AIRFLOW-6500
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6500
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, scheduler
>Affects Versions: 1.10.6
>Reporter: Pavan Maguluri
>Priority: Major
>
> I have the below Dag with "schedule_interval" set to None and "start_date" 
> set to airflow.utils.dates.days_ago(1). 
> Task1 --> Task2 --> End
> When I triggered the Dag from the Web-UI it is failing with the below reason. 
> Other Dags which has "schedule_interval" set to some value don't have this 
> issue.
> *[2020-01-07 07:06:39,962] \{taskinstance.py:624} INFO - Dependencies not met 
> for , 
> dependency 'Execution Date' FAILED: Execution date 
> 2020-01-07T07:06:58.550778+00:00 is in the future (the current date is 
> 2020-01-07T07:06:39.962619+00:00).*
>  
> Default Pipeline Arguments: 
> {code:python}
> PIPELINE_DEFAULT_ARGS = {
>  'owner': 'owner',
>  'start_date': airflow.utils.dates.days_ago(1),
>  'depends_on_past': True,
>  'email_on_failure': False,
>  'email_on_retry': False,
> }{code}
> All the solutions that I have read online tags with schedule_interval set to 
> some value and start_date being dynamic (like datetime.now()). I don't have 
> either of those settings, but, still see the problem. 
> Appreciate if you can shed some light to fix this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tooptoop4 commented on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

[GitBox]

tooptoop4 commented on issue #7038: [AIRFLOW-4495] allow externally triggered 
dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-571797817
 
 
   passed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

[GitBox]

codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally 
triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@b5bd9ab`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `96.9%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7038/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#7038   +/-   ##
   =
 Coverage  ?   84.87%   
   =
 Files ?  680   
 Lines ?38828   
 Branches  ?0   
   =
 Hits  ?32955   
 Misses? 5873   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/prev\_dagrun\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcHJldl9kYWdydW5fZGVwLnB5)
 | `80.55% <ø> (ø)` | |
   | 
[...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5)
 | `91.42% <ø> (ø)` | |
   | 
[airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5)
 | `98.61% <ø> (ø)` | |
   | 
[airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5)
 | `85.96% <ø> (ø)` | |
   | 
[...ib/operators/azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9jb250YWluZXJfaW5zdGFuY2VzX29wZXJhdG9yLnB5)
 | `78.14% <ø> (ø)` | |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.25% <ø> (ø)` | |
   | 
[airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=)
 | `52.5% <0%> (ø)` | |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `89.74% <100%> (ø)` | |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `91.4% <100%> (ø)` | |
   | 
[airflow/api/common/experimental/trigger\_dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC90cmlnZ2VyX2RhZy5weQ==)
 | `97.95% <100%> (ø)` | |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). 
Last update 
[b5bd9ab...e5547e9](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

[GitBox]

codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally 
triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) 
Report
   > Merging 
[#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc)
 will **decrease** coverage by `0.76%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7038/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7038  +/-   ##
   ==
   - Coverage   85.06%   84.29%   -0.77% 
   ==
 Files 680  680  
 Lines   3875738828  +71 
   ==
   - Hits3296832731 -237 
   - Misses   5789 6097 +308
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `88.4% <100%> (-0.44%)` | :arrow_down: |
   | 
[airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5)
 | `90.67% <100%> (+0.05%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | ... and [22 
more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). 
Last update 
[41a9bfd...e5547e9](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

[GitBox]

codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally 
triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) 
Report
   > Merging 
[#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc)
 will **decrease** coverage by `0.76%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7038/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7038  +/-   ##
   ==
   - Coverage   85.06%   84.29%   -0.77% 
   ==
 Files 680  680  
 Lines   3875738828  +71 
   ==
   - Hits3296832731 -237 
   - Misses   5789 6097 +308
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `88.4% <100%> (-0.44%)` | :arrow_down: |
   | 
[airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5)
 | `90.67% <100%> (+0.05%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | ... and [22 
more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). 
Last update 
[41a9bfd...e5547e9](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator

[GitBox]

m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement 
LatestOnlyOperator as BaseBranchOperator
URL: https://github.com/apache/airflow/pull/5970#issuecomment-571794871
 
 
   Documentation updated and pylint issues fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-3435) Example KubernetesOperator docs for Volume mount wrong

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik closed AIRFLOW-3435.
---
Resolution: Implemented

Resolved in https://github.com/apache/airflow/pull/3875

> Example KubernetesOperator docs for Volume mount wrong
> --
>
> Key: AIRFLOW-3435
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3435
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: documentation, operators
>Affects Versions: 1.10.1
> Environment: Airflow 1.10.1 on Docker, Kubernetes running on minikube 
> v0.28.2, kubernetes client version: 1.12.3, kubernetes server version: 
> 1.10.0, python version of airflow 3.6
>Reporter: Sai Varun Reddy Daram
>Assignee: Daniel Imberman
>Priority: Major
>  Labels: kubernetes
>
> I have followed the standard example describe here 
> [https://airflow.apache.org/kubernetes.html#kubernetes-operator,] and I'm 
> getting this error.
> {{Traceback (most recent call last): File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
>  line 55, in run_pod_async resp = 
> self._client.create_namespaced_pod(body=req, namespace=pod.namespace) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
>  line 6115, in create_namespaced_pod (data) = 
> self.create_namespaced_pod_with_http_info(namespace, body, **kwargs) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
>  line 6206, in create_namespaced_pod_with_http_info 
> collection_formats=collection_formats) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 321, in call_api _return_http_data_only, collection_formats, 
> _preload_content, _request_timeout) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 155, in __call_api _request_timeout=_request_timeout) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 364, in request body=body) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 266, 
> in POST body=body) File 
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, 
> in request raise ApiException(http_resp=r) 
> kubernetes.client.rest.ApiException: (422) Reason: Unprocessable Entity HTTP 
> response headers: HTTPHeaderDict(\{'Content-Type': 'application/json', 
> 'Date': 'Mon, 03 Dec 2018 07:16:10 GMT', 'Content-Length': '393'}) HTTP 
> response body: 
> \{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod
>  \"test-4f6c8ada\" is invalid: spec.containers[0].volumeMounts[1].name: Not 
> found: 
> \"test-volume\"","reason":"Invalid","details":\{"name":"test-4f6c8ada","kind":"Pod","causes":[{"reason":"FieldValueNotFound","message":"Not
>  found: 
> \"test-volume\"","field":"spec.containers[0].volumeMounts[1].name"}]},"code":422}}}
>  
>  
> {{The code is:}}
> {code:java}
> // code placeholder
> from airflow.contrib.kubernetes.volume import Volume
> from airflow.contrib.kubernetes.volume_mount import VolumeMount
> from airflow.contrib.operators.kubernetes_pod_operator import 
> KubernetesPodOperator
> from airflow.contrib.kubernetes.secret import Secret
> from airflow import DAG
> from datetime import datetime, timedelta
> current_date = datetime.utcnow()
> default_args = {
> 'owner': 'root',
> 'depends_on_past': False,
> 'retries': 1,
> 'retry_delay': timedelta(minutes=1),
> }
> secret_file = Secret('volume', '/etc/sql_conn', 'airflow-secrets', 
> 'sql_alchemy_conn')
> secret_env = Secret('env', 'SQL_CONN', 'airflow-secrets', 'sql_alchemy_conn')
> volume_mount = VolumeMount('test-volume',
> mount_path='/root/mount_file',
> sub_path=None,
> read_only=True)
> volume_config = {
> 'persistentVolumeClaim':
> {
> 'claimName': 'test-volume'
> }
> }
> volume = Volume(name='test-volume', configs=volume_config)
> with DAG(
> dag_id='MMM_DAG', default_args=default_args,
> start_date=current_date,
> concurrency=1,
> schedule_interval=None) as d:
> k = KubernetesPodOperator(namespace='default',
> image="ubuntu:16.04",
> cmds=["bash", "-cx"],
> arguments=["echo", "10"],
> in_cluster=True,
> labels={"foo": "bar"},
> secrets=[secret_file, secret_env],
> volume=[volume],
> volume_mounts=[volume_mount],
> name="test",
> task_id="task",
> is_delete_operator_pod=True,
> hostnetwork=False
> )
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-915) Allow task context to be modified by pre_execute hook

[Paul Rhodes (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010107#comment-17010107
 ] 

Paul Rhodes commented on AIRFLOW-915:
-

This is no longer required since the it can be achieved by overriding the 
render_template_fields method in the subclass, as the example below:
{code:java}
def render_template_fields(self, context, jinja_env=None):
super().render_template_fields(context, jinja_env)
context['params']['dynamic_param'] = 
myfunction(context['execution_date'])
 {code}

> Allow task context to be modified by pre_execute hook
> -
>
> Key: AIRFLOW-915
> URL: https://issues.apache.org/jira/browse/AIRFLOW-915
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.8.0
>Reporter: Jeremiah Lowin
>Priority: Minor
>
> Currently, Operators do two things that prevent the pre_execute hook from 
> modifying the jinja context:
> 1. the Operator templates are rendered immediately BEFORE calling pre_execute
> 2. even though a context was already generated for the operator, the context 
> is regenerated for template rendering, meaning that modifications to the 
> operator context wouldn't matter anyway.
> The proper course of events should be:
> 1. generate operator context
> 2. pass context to pre_execute where it could (potentially) be modified
> 3. use that context to render operator templates



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-915) Allow task context to be modified by pre_execute hook

[Paul Rhodes (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rhodes reassigned AIRFLOW-915:
---

Assignee: Paul Rhodes

> Allow task context to be modified by pre_execute hook
> -
>
> Key: AIRFLOW-915
> URL: https://issues.apache.org/jira/browse/AIRFLOW-915
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.8.0
>Reporter: Jeremiah Lowin
>Assignee: Paul Rhodes
>Priority: Minor
>
> Currently, Operators do two things that prevent the pre_execute hook from 
> modifying the jinja context:
> 1. the Operator templates are rendered immediately BEFORE calling pre_execute
> 2. even though a context was already generated for the operator, the context 
> is regenerated for template rendering, meaning that modifications to the 
> operator context wouldn't matter anyway.
> The proper course of events should be:
> 1. generate operator context
> 2. pass context to pre_execute where it could (potentially) be modified
> 3. use that context to render operator templates



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6975: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need)

[GitBox]

codecov-io edited a comment on issue #6975: [AIRFLOW-1467] Dynamic pooling via 
allowing tasks to use more than one pool slot (depending upon the need)
URL: https://github.com/apache/airflow/pull/6975#issuecomment-569916253
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=h1) 
Report
   > Merging 
[#6975](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/da2a617cb7494c6e358b079a6cf8c4129e742a63?src=pr=desc)
 will **increase** coverage by `0.08%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6975/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #6975  +/-   ##
   =
   + Coverage   84.81%   84.9%   +0.08% 
   =
 Files 679 680   +1 
 Lines   38491   38903 +412 
   =
   + Hits32648   33032 +384 
   - Misses   58435871  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.26% <ø> (+0.18%)` | :arrow_up: |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <ø> (ø)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `95% <100%> (+0.08%)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.75% <0%> (-20%)` | :arrow_down: |
   | 
[airflow/utils/file.py](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9maWxlLnB5)
 | `86.3% <0%> (-5.17%)` | :arrow_down: |
   | ... and [97 
more](https://codecov.io/gh/apache/airflow/pull/6975/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=footer). 
Last update 
[da2a617...a51783b](https://codecov.io/gh/apache/airflow/pull/6975?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook 
resolve connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571765352
 
 
   existing tests for connection added via db/cli needs to work


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve 
connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571765352
 
 
   existing tests for connection added via/cli needs to work


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on a change in pull request #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

tooptoop4 commented on a change in pull request #7075: [AIRFLOW-6212] 
SparkSubmitHook resolve connection
URL: https://github.com/apache/airflow/pull/7075#discussion_r363944283
 
 

 ##
 File path: airflow/contrib/hooks/spark_submit_hook.py
 ##
 @@ -190,17 +189,22 @@ def _resolve_connection(self):
 # Master can be local, yarn, spark://HOST:PORT, mesos://HOST:PORT 
and
 # k8s://https://:
 conn = self.get_connection(self._conn_id)
-if conn.port:
-conn_data['master'] = "{}:{}".format(conn.host, conn.port)
+if conn.conn_type in ['spark', 'mesos']:
 
 Review comment:
   revert this section 192-200


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on a change in pull request #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

tooptoop4 commented on a change in pull request #7075: [AIRFLOW-6212] 
SparkSubmitHook resolve connection
URL: https://github.com/apache/airflow/pull/7075#discussion_r363943123
 
 

 ##
 File path: airflow/contrib/hooks/spark_submit_hook.py
 ##
 @@ -174,8 +174,7 @@ def _resolve_should_track_driver_status(self):
 subsequent spark-submit status requests after the initial spark-submit 
request
 :return: if the driver status should be tracked
 """
-return ('spark://' in self._connection['master'] and
-self._connection['deploy_mode'] == 'cluster')
+return self._connection['deploy_mode'] == 'cluster'
 
 Review comment:
   return (('spark://' in self._connection['master'] or conn_type starts with 
spark) and self._connection['deploy_mode'] == 'cluster')


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

[GitBox]

tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve 
connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571762855
 
 
   > > > @tooptoop4 if we don't remove the spark check on line 177, how to use 
this hook to track driver status deployed on yarn, mesos, or k8s? Since I think 
`spark://` is only for standalone mode.
   > > > Or this hook is created only for standalone mode?
   > > 
   > > 
   > > yes.there is no concept of async driver status poll for other modes , 
read https://spark.apache.org/docs/latest/running-on-yarn.html ! in other modes 
the submit to launch is synchronous . i think u can cancel this @albertusk95
   > 
   > I couldn't find any info stating that there's no async driver polling for 
YARN anyway from the provided link.
   
   There isn't async driver polling in YARN, I know Spark on YARN.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services