date:20191111

BasPH commented on a change in pull request #6549: [AIRFLOW-5880] Enforce 
unique task ids
URL: https://github.com/apache/airflow/pull/6549#discussion_r345057908
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -1190,13 +1189,8 @@ def add_task(self, task):
 task.end_date = min(task.end_date, self.end_date)
 
 if task.task_id in self.task_dict:
 
 Review comment:
   IMO it should. I often see it going wrong with for loops where people forget 
to make unique task ids. E.g.:
   
   ```python
   for i in range(5):
   DummyOperator(task_id="task")
   ```
   
   In this case, currently no warning/error is given an only the last defined 
task is saved. I think enforcing unique task ids here would be a good thing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

potiuk edited a comment on issue #6536: [AIRFLOW-5885] List of tests is 
generated by pre-commit not Docker build
URL: https://github.com/apache/airflow/pull/6536#issuecomment-552773959
 
 
   Side comment: I think eventually everyone should use pre-commit. It makes it 
so much easier to keep the source code in order and to fix many of the problems 
reported by Travis even before they are reported. We should strongly encourage 
people to use it. I think we should encourage good practices rather than 
discourage them.
   
   But I agree having the .txt file in the repo is kind of superfluous. I think 
it is a good idea to generate it when you enter Breeze. I can run it in 
background.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

potiuk commented on issue #6536: [AIRFLOW-5885] List of tests is generated by 
pre-commit not Docker build
URL: https://github.com/apache/airflow/pull/6536#issuecomment-552773959
 
 
   Side comment: I think eventually everyone should use pre-commit. It makes it 
so much easier to keep the source code in order and to fix many of the problems 
reported by Travis even before they are reported. We should strongly encourage 
people to use it. I think we should encourage good practices rather than 
discourage them.
   
   But I agree having the .txt file in the repo is kind of superfluous. I think 
it is a good idea to generate it when you enter Breeze. I can run it in 
background when I will change it :).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6547: [AIRFLOW-5898] fix alembic crash due to typing import

potiuk edited a comment on issue #6547: [AIRFLOW-5898] fix alembic crash due to 
typing import
URL: https://github.com/apache/airflow/pull/6547#issuecomment-552771864
 
 
   I think "custom_typing" is not best name. Maybe "typing_compat" might 
reflect its purpose better?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6547: [AIRFLOW-5898] fix alembic crash due to typing import

potiuk commented on issue #6547: [AIRFLOW-5898] fix alembic crash due to typing 
import
URL: https://github.com/apache/airflow/pull/6547#issuecomment-552771864
 
 
   I think "custom_typing" is not best name. Maybe "typing_compat" might 
reflect its purpose better.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6512: [AIRFLOW-5824] Added AWS DataSync Hook and Operator

codecov-io edited a comment on issue #6512: [AIRFLOW-5824] Added AWS DataSync 
Hook and Operator
URL: https://github.com/apache/airflow/pull/6512#issuecomment-551648543
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=h1) 
Report
   > Merging 
[#6512](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `0.77%`.
   > The diff coverage is `23.25%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6512/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6512  +/-   ##
   ==
   - Coverage   84.07%   83.29%   -0.78% 
   ==
 Files 639  642   +3 
 Lines   3690037201 +301 
   ==
   - Hits3102430987  -37 
   - Misses   5876 6214 +338
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/aws\_datasync\_hook.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19kYXRhc3luY19ob29rLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/providers/amazon/aws/hooks/datasync.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9kYXRhc3luYy5weQ==)
 | `17.55% <17.55%> (ø)` | |
   | 
[airflow/providers/amazon/aws/operators/datasync.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvZGF0YXN5bmMucHk=)
 | `25.9% <25.9%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=footer). 
Last update 
[d25ef2b...fe37ffa](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6512: [AIRFLOW-5824] Added AWS DataSync Hook and Operator

codecov-io edited a comment on issue #6512: [AIRFLOW-5824] Added AWS DataSync 
Hook and Operator
URL: https://github.com/apache/airflow/pull/6512#issuecomment-551648543
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=h1) 
Report
   > Merging 
[#6512](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `0.77%`.
   > The diff coverage is `23.25%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6512/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6512  +/-   ##
   ==
   - Coverage   84.07%   83.29%   -0.78% 
   ==
 Files 639  642   +3 
 Lines   3690037201 +301 
   ==
   - Hits3102430987  -37 
   - Misses   5876 6214 +338
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/aws\_datasync\_hook.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19kYXRhc3luY19ob29rLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/providers/amazon/aws/hooks/datasync.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9kYXRhc3luYy5weQ==)
 | `17.55% <17.55%> (ø)` | |
   | 
[airflow/providers/amazon/aws/operators/datasync.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvZGF0YXN5bmMucHk=)
 | `25.9% <25.9%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/6512/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=footer). 
Last update 
[d25ef2b...fe37ffa](https://codecov.io/gh/apache/airflow/pull/6512?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from tests/core.py

potiuk commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from 
tests/core.py
URL: https://github.com/apache/airflow/pull/6544#issuecomment-552768400
 
 
   I am sure it's not every time. It never ever happened to me. I am super 
surprised because exactly the same check is run on Travis as in your local 
environment. 
   
   I suggest you enable it and let me know when it happens again. I am happy to 
take a look when it happens again and help to find the reason (if this is 
really so frequent). You are the only person I know who has this problem so it 
would be worth checking what's wrong in your setup and prevent it happening 
also for others.
   
   If you look now at Travis - there is an over-indent in test_hdfs which would 
have been detected locally if you have pre-commits enabled. That's the whole 
point of pre-commit.
   
   Re. Import above - can you please explain what was wrong? The change above 
is not pre-commit - but simply isort in the works. What's wrong about this? The 
isort sorting algorithm is such that the "airflow" (my) imports are always at 
the end.  We all agreed we use isort so we should follow it.
   Was it failing in Travis static checks or unit tests? If the latter and 
there is a good reason to keep sorting, then it should be as simple as 
disabling isort for the affected line or even for the whole file: 
https://isort.readthedocs.io/en/latest/#skip-processing-of-imports-outside-of-configuration
   
   Or we can configure skip rules in setup.cfg
   
   Let me know if any of those help. And happy to help further.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4593) Memory leak in Airflow scheduler

2019-11-11 Thread yuansicheng (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972109#comment-16972109
 ] 

yuansicheng commented on AIRFLOW-4593:
--

I find this problem in version 1.10.6 .  I run the scheduler in a kubenetes 
pod, the memory usage is growing about 100M per day

And when I use the airflow 1.10.4 to run the scheduler, the problem has not 
gone away too.

> Memory leak in Airflow scheduler
> 
>
> Key: AIRFLOW-4593
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4593
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Nikhil SInghal
>Priority: Major
> Attachments: Screenshot 2019-05-30 at 3.19.33 PM.png
>
>
> We are running Apache Airflow on Kubernetes. When I see my Grafana Dashboard 
> I see that the memory used is consistently increasing. Can anyone give me 
> some pointers for how can I debug this or with existing issues/solutions 
> related to this.
> This is a sharp increase in Airflow scheduler and slow increase for worker 
> and webserver
> !Screenshot 2019-05-30 at 3.19.33 PM.png|width=100%!
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] zhongjiajie commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate

zhongjiajie commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and 
remove duplicate
URL: https://github.com/apache/airflow/pull/6487#issuecomment-552748900
 
 
   @mik-laj I rebase on master, PTAL, if you have time. I think the CI will 
pass cause this PR only doc change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] baolsen commented on issue #6512: [AIRFLOW-5824] Added AWS DataSync Hook and Operator

baolsen commented on issue #6512: [AIRFLOW-5824] Added AWS DataSync Hook and 
Operator
URL: https://github.com/apache/airflow/pull/6512#issuecomment-552745857
 
 
   Hi @potiuk my changes are done. Ready for review / merge when you are.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6531: [AIRFLOW-5882] Add ti_dep for not being in the RUNNING state

codecov-io edited a comment on issue #6531: [AIRFLOW-5882] Add ti_dep for not 
being in the RUNNING state
URL: https://github.com/apache/airflow/pull/6531#issuecomment-552724876
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=h1) 
Report
   > Merging 
[#6531](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `94.73%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6531/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6531  +/-   ##
   ==
   - Coverage   84.07%   83.78%   -0.29% 
   ==
 Files 639  641   +2 
 Lines   3690036919  +19 
   ==
   - Hits3102430934  -90 
   - Misses   5876 5985 +109
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvX19pbml0X18ucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/task\_not\_running\_dep.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdGFza19ub3RfcnVubmluZ19kZXAucHk=)
 | `94.11% <94.11%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.15% <0%> (-0.33%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=footer). 
Last update 
[d25ef2b...9bf9f42](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6531: [AIRFLOW-5882] Add ti_dep for not being in the RUNNING state

codecov-io commented on issue #6531: [AIRFLOW-5882] Add ti_dep for not being in 
the RUNNING state
URL: https://github.com/apache/airflow/pull/6531#issuecomment-552724876
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=h1) 
Report
   > Merging 
[#6531](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `94.73%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6531/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6531  +/-   ##
   ==
   - Coverage   84.07%   83.78%   -0.29% 
   ==
 Files 639  641   +2 
 Lines   3690036919  +19 
   ==
   - Hits3102430934  -90 
   - Misses   5876 5985 +109
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/ti\_deps/deps/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvX19pbml0X18ucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/ti\_deps/deps/task\_not\_running\_dep.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdGFza19ub3RfcnVubmluZ19kZXAucHk=)
 | `94.11% <94.11%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.15% <0%> (-0.33%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6531/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=footer). 
Last update 
[d25ef2b...9bf9f42](https://codecov.io/gh/apache/airflow/pull/6531?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6545: [AIRFLOW-5896] Move email stuff from tests/core.py

codecov-io commented on issue #6545: [AIRFLOW-5896] Move email stuff from 
tests/core.py
URL: https://github.com/apache/airflow/pull/6545#issuecomment-552723504
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=h1) 
Report
   > Merging 
[#6545](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/3fa64ea7a9476de4810f3b95f170074a2b422206?src=pr=desc)
 will **decrease** coverage by `0.06%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6545/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6545  +/-   ##
   ==
   - Coverage   83.83%   83.76%   -0.07% 
   ==
 Files 638  639   +1 
 Lines   3686136900  +39 
   ==
   + Hits3090230909   +7 
   - Misses   5959 5991  +32
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `96.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/pagerduty\_hook.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3BhZ2VyZHV0eV9ob29rLnB5)
 | `81.57% <0%> (ø)` | |
   | 
[airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5)
 | `95.78% <0%> (+0.04%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.31% <0%> (+0.16%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.52% <0%> (+0.84%)` | :arrow_up: |
   | ... and [5 
more](https://codecov.io/gh/apache/airflow/pull/6545/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=footer). 
Last update 
[3fa64ea...f6682d9](https://codecov.io/gh/apache/airflow/pull/6545?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group tests for the Dags command

codecov-io commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group 
tests for the Dags command
URL: https://github.com/apache/airflow/pull/6543#issuecomment-552719983
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=h1) 
Report
   > Merging 
[#6543](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/3fa64ea7a9476de4810f3b95f170074a2b422206?src=pr=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6543/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6543  +/-   ##
   ==
   - Coverage   83.83%   83.77%   -0.06% 
   ==
 Files 638  639   +1 
 Lines   3686136900  +39 
   ==
   + Hits3090230913  +11 
   - Misses   5959 5987  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `96.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/pagerduty\_hook.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3BhZ2VyZHV0eV9ob29rLnB5)
 | `81.57% <0%> (ø)` | |
   | 
[airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5)
 | `95.78% <0%> (+0.04%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.31% <0%> (+0.16%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.52% <0%> (+0.84%)` | :arrow_up: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/6543/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=footer). 
Last update 
[3fa64ea...c31c5ce](https://codecov.io/gh/apache/airflow/pull/6543?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate

mik-laj commented on a change in pull request #6487: [AIRFLOW-XXX] Alphabetical 
table and remove duplicate
URL: https://github.com/apache/airflow/pull/6487#discussion_r345005099
 
 

 ##
 File path: docs/operators-and-hooks-ref.rst
 ##
 @@ -671,6 +619,54 @@ These integrations allow you to perform various 
operations within the Google Clo
  - :mod:`airflow.providers.google.cloud.operators.vision`
  -
 
+   * - `Compute Engine `__
 
 Review comment:
   The script will provide an additional option, but by default the tables 
should also be sorted


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate

mik-laj commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove 
duplicate
URL: https://github.com/apache/airflow/pull/6487#issuecomment-552719533
 
 
   Sorry. I was busy lately because I was working on a new website. Can you do 
a rebase? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

mik-laj commented on a change in pull request #6538: [AIRFLOW-5886] Selective 
copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]
URL: https://github.com/apache/airflow/pull/6538#discussion_r345003999
 
 

 ##
 File path: Dockerfile
 ##
 @@ -329,36 +328,45 @@ COPY airflow/www/ ${AIRFLOW_SOURCES}/airflow/www/
 # Package NPM for production
 RUN npm run prod
 
-# Cache for this line will be automatically invalidated if any
-# of airflow sources change
-COPY . ${AIRFLOW_SOURCES}/
-
-WORKDIR ${AIRFLOW_SOURCES}
-
-# Finally install the requirements from the latest sources
-RUN pip install -e ".[${AIRFLOW_EXTRAS}]"
-
-# Additional python deps to install
-ARG ADDITIONAL_PYTHON_DEPS=""
-
-RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
-pip install ${ADDITIONAL_PYTHON_DEPS}; \
-fi
-
 COPY ./scripts/docker/entrypoint.sh /entrypoint.sh
 
 COPY .bash_completion run-tests-complete run-tests ${HOME}/
+
 COPY .bash_completion.d/run-tests-complete \
  ${HOME}/.bash_completion.d/run-tests-complete
 
-COPY license-templates ${AIRFLOW_SOURCES}/license-templates
-
 RUN echo ". ${HOME}/.bash_completion" >> "${HOME}/.bashrc"
 
 RUN chmod +x "${HOME}/run-tests-complete"
 
 RUN chmod +x "${HOME}/run-tests"
 
+# Copy selected subdirectories only
+COPY .github/ ${AIRFLOW_SOURCES}/.github/
+COPY dev/ ${AIRFLOW_SOURCES}/dev/
 
 Review comment:
   Do we need this folder in the image?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

mik-laj commented on a change in pull request #6538: [AIRFLOW-5886] Selective 
copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]
URL: https://github.com/apache/airflow/pull/6538#discussion_r345003935
 
 

 ##
 File path: Dockerfile
 ##
 @@ -329,36 +328,45 @@ COPY airflow/www/ ${AIRFLOW_SOURCES}/airflow/www/
 # Package NPM for production
 RUN npm run prod
 
-# Cache for this line will be automatically invalidated if any
-# of airflow sources change
-COPY . ${AIRFLOW_SOURCES}/
-
-WORKDIR ${AIRFLOW_SOURCES}
-
-# Finally install the requirements from the latest sources
-RUN pip install -e ".[${AIRFLOW_EXTRAS}]"
-
-# Additional python deps to install
-ARG ADDITIONAL_PYTHON_DEPS=""
-
-RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
-pip install ${ADDITIONAL_PYTHON_DEPS}; \
-fi
-
 COPY ./scripts/docker/entrypoint.sh /entrypoint.sh
 
 COPY .bash_completion run-tests-complete run-tests ${HOME}/
+
 COPY .bash_completion.d/run-tests-complete \
  ${HOME}/.bash_completion.d/run-tests-complete
 
-COPY license-templates ${AIRFLOW_SOURCES}/license-templates
-
 RUN echo ". ${HOME}/.bash_completion" >> "${HOME}/.bashrc"
 
 RUN chmod +x "${HOME}/run-tests-complete"
 
 RUN chmod +x "${HOME}/run-tests"
 
+# Copy selected subdirectories only
+COPY .github/ ${AIRFLOW_SOURCES}/.github/
 
 Review comment:
   Do we need this folder in the image?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-5903) Remove serve_logs command



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula closed AIRFLOW-5903.
--
Resolution: Fixed

> Remove serve_logs command
> -
>
> Key: AIRFLOW-5903
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5903
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6551: [AIRFLOW-5900] avoid unnecessary system calls in heartbeat_callback

codecov-io commented on issue #6551: [AIRFLOW-5900] avoid unnecessary system 
calls in heartbeat_callback
URL: https://github.com/apache/airflow/pull/6551#issuecomment-552715725
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=h1) 
Report
   > Merging 
[#6551](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6551/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6551  +/-   ##
   ==
   - Coverage   84.07%   83.78%   -0.29% 
   ==
 Files 639  639  
 Lines   3690036900  
   ==
   - Hits3102430917 -107 
   - Misses   5876 5983 +107
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `90% <100%> (+5%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6551/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=footer). 
Last update 
[d25ef2b...064d1fc](https://codecov.io/gh/apache/airflow/pull/6551?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on issue #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric

houqp commented on issue #6342: [AIRFLOW-5662] fix incorrect naming for 
scheduler used slot metric
URL: https://github.com/apache/airflow/pull/6342#issuecomment-552706447
 
 
   After thinking more about it, I now think it's better to approach this with 
a bigger refactoring. Let me know what you think about what I proposed below:
   
   Currently, in `_find_executable_task_instances` loop, for metrics collection 
purpose, we issue one sql query and multiple statsds calls per pool. We can 
move all these instrumentation overhead into the end of the loop in 
`_execute_helper()` in scheduler_job.py. That way, we get the following 
benefits:
   
   1. the actual scheduling logic can finish as fast as it can without having 
to wait for metrics collection code, which has nothing to do with the 
scheduling logic itself. we can even run the metrics collection code in a 
separate loop in a separate thread to reduce the overhead to minimal if needed.
   2. the code is more decoupled and readable by separating out the metrics 
collection code from the actual scheduling logic.
   3. if we collect metrics for all pools in one place instead through a loop 
iteration, we can get all the stats we need efficiently from DB in just one SQL 
query.
   
   As for what's the right value to use to report, I think we should settle 
down on only reporting committed values whenever appropriate. i.e. metrics 
should be calculated using data from persistent datastore. For example, if a 
variable is updated in memory, but not committed to DB, it shouldn't be 
reported. Because a crash before the commit could invalidate the metric. There 
are rare cases where we do want to report uncommitted value as metric, but 
usually those are for troubleshooting purposes, which shouldn't apply to the 
metrics we are dealing with here.
   
   For the in memory open slot metric specifically, in 
`_find_executable_task_instances`, the task instance hasn't been scheduled to 
run yet, so it would be misleading to report the in memory value of the new 
open slot count as metric, it is only used as an internal counter to decide how 
many tasks to schedule. If the scheduler crashed before it was able to schedule 
those prepared tasks and commit to DB, we should make sure those in memory 
states are not reported, because a later restart won't resume from it.
   
   By moving the metrics collection code out of 
`_find_executable_task_instances` and into `_execute_helper`, we won't have to 
worry about which value to pick to report, we will just report what's been 
committed in the database.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dstandish commented on issue #6426: [AIRFLOW-5751] add get_uri method to Connection

dstandish commented on issue #6426: [AIRFLOW-5751] add get_uri method to 
Connection
URL: https://github.com/apache/airflow/pull/6426#issuecomment-552705561
 
 
   @ashb @mik-laj friendly nag
   
   this can help people produce URI reliably


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6550: [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver

codecov-io commented on issue #6550: [AIRFLOW-5897] Allow setting -1 as pool 
slots value in webserver
URL: https://github.com/apache/airflow/pull/6550#issuecomment-552704856
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=h1) 
Report
   > Merging 
[#6550](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6550/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6550  +/-   ##
   ==
   - Coverage   84.07%   84.07%   -0.01% 
   ==
 Files 639  639  
 Lines   3690036900  
   ==
   - Hits3102431022   -2 
   - Misses   5876 5878   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6550/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.47% <ø> (ø)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6550/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.15% <0%> (-0.33%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=footer). 
Last update 
[d25ef2b...544c8c5](https://codecov.io/gh/apache/airflow/pull/6550?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric

houqp commented on a change in pull request #6342: [AIRFLOW-5662] fix incorrect 
naming for scheduler used slot metric
URL: https://github.com/apache/airflow/pull/6342#discussion_r344991021
 
 

 ##
 File path: airflow/jobs/scheduler_job.py
 ##
 @@ -982,10 +982,13 @@ def _find_executable_task_instances(self, 
simple_dag_bag, states, session=None):
 
 
Stats.gauge('pool.starving_tasks.{pool_name}'.format(pool_name=pool_name),
 num_starving_tasks)
+slots_stats = pools[pool_name].slots_stats()
 
Stats.gauge('pool.open_slots.{pool_name}'.format(pool_name=pool_name),
-pools[pool_name].open_slots())
+slots_stats["open"])
 
 Review comment:
   yes, db hasn't been updated in DB at this point yet. i will reply to this in 
my follow up comment below as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5903) Remove serve_logs command

Kamil Bregula created AIRFLOW-5903:
--

 Summary: Remove serve_logs command
 Key: AIRFLOW-5903
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5903
 Project: Apache Airflow
  Issue Type: Improvement
  Components: cli
Affects Versions: 1.10.6
Reporter: Kamil Bregula






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] akki edited a comment on issue #5489: [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator)

akki edited a comment on issue #5489: [AIRFLOW-4843] Allow orchestration via 
Docker Swarm (SwarmOperator)
URL: https://github.com/apache/airflow/pull/5489#issuecomment-552699552
 
 
   @hredestig You can find the PR here - 
https://github.com/apache/airflow/pull/6552. Please see if it looks good to you.
   
   Also, for anyone wondering what this feature/commit is about, I wrote a 
little bit here - 
https://medium.com/analytics-vidhya/orchestrating-airflow-tasks-with-docker-swarm-69b5fb2723a7.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] akki commented on issue #5489: [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator)

akki commented on issue #5489: [AIRFLOW-4843] Allow orchestration via Docker 
Swarm (SwarmOperator)
URL: https://github.com/apache/airflow/pull/5489#issuecomment-552699552
 
 
   @hredestig You can find the PR here - 
https://github.com/apache/airflow/pull/6552. Please consider helping with your 
reviews/testing.
   
   Also, for anyone wondering what this feature/commit is about, I wrote a 
little bit here - 
https://medium.com/analytics-vidhya/orchestrating-airflow-tasks-with-docker-swarm-69b5fb2723a7.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] akki commented on issue #6542: AIRFLOW-5854: Add support for `tty` parameter in Docker related operators

akki commented on issue #6542: AIRFLOW-5854: Add support for `tty` parameter in 
Docker related operators
URL: https://github.com/apache/airflow/pull/6542#issuecomment-552698656
 
 
   Thank you very much for the quick review! :)
   
   Cherry-picked as 
https://github.com/apache/airflow/commit/10534a1044d8660dcaa86695fdde1ffeaa8829f7
 into `v1-10-test`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5901) Group tests for the tasks and variables command



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971977#comment-16971977
 ] 

ASF GitHub Bot commented on AIRFLOW-5901:
-

mik-laj commented on pull request #6554:  [AIRFLOW-5901][depends on 
AIRFLOW-5894] Group tests for the tasks and variables command
URL: https://github.com/apache/airflow/pull/6554
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Group tests for the tasks and variables command
> ---
>
> Key: AIRFLOW-5901
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5901
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971976#comment-16971976
 ] 

ASF GitHub Bot commented on AIRFLOW-5902:
-

houqp commented on pull request #6553: [AIRFLOW-5902] avoid unnecessary sleep 
to maintain local task job heart rate
URL: https://github.com/apache/airflow/pull/6553
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   After a heartbeat, time since last hearbeat will always be lower than heart 
rate, so the subsequent sleep is unnecessary. It also spams the task log with 
lots of info messages.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> local_task_job is sleeping more often than it needs and spamming task log
> -
>
> Key: AIRFLOW-5902
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5902
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> local_task_log's _execute loop is calling extra unnecessary sleep after a 
> heartbeat to keep up with the heart rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] houqp opened a new pull request #6553: [AIRFLOW-5902] avoid unnecessary sleep to maintain local task job heart rate

houqp opened a new pull request #6553: [AIRFLOW-5902] avoid unnecessary sleep 
to maintain local task job heart rate
URL: https://github.com/apache/airflow/pull/6553
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   After a heartbeat, time since last hearbeat will always be lower than heart 
rate, so the subsequent sleep is unnecessary. It also spams the task log with 
lots of info messages.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6554: [AIRFLOW-5901][depends on AIRFLOW-5894] Group tests for the tasks and variables command

mik-laj opened a new pull request #6554:  [AIRFLOW-5901][depends on 
AIRFLOW-5894] Group tests for the tasks and variables command
URL: https://github.com/apache/airflow/pull/6554
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log

QP Hou created AIRFLOW-5902:
---

 Summary: local_task_job is sleeping more often than it needs and 
spamming task log
 Key: AIRFLOW-5902
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5902
 Project: Apache Airflow
  Issue Type: Bug
  Components: core
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


local_task_log's _execute loop is calling extra unnecessary sleep after a 
heartbeat to keep up with the heart rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5850) Make DockerSwarmOperator capture task logs



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971973#comment-16971973
 ] 

ASF GitHub Bot commented on AIRFLOW-5850:
-

akki commented on pull request #6552: AIRFLOW-5850: Capture task logs in 
DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5850
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit test:
   `test_logging`
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make DockerSwarmOperator capture task logs
> --
>
> Key: AIRFLOW-5850
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5850
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: contrib, operators
>Affects Versions: 1.10.6
> Environment: Reproducible everywhere since missing feature - Tested 
> via puckel/airflow:latest
>Reporter: Henning Redestig
>Assignee: Akshesh Doshi
>Priority: Major
>  Labels: docker, orchestration, swarm
> Fix For: 1.10.7
>
>
> contrib.operators.DockerSwarmOperator gives output on task starting and 
> exiting but does not capture the logs from the task.  E.g.
> [2019-11-04 16:27:30,337] \{{docker_swarm_operator.py:125}} INFO - Service 
> started: \{'ID': 'mxjn03sm32kfm8bcczs4qw6tu'}
> \## the task outputs logs here but they do not show up in the webapp..
> [2019-11-04 16:27:32,909] \{{docker_swarm_operator.py:136}} INFO - Service 
> status before exiting: complete
> [2019-11-04 16:27:37,298] \{{logging_mixin.py:95}} INFO - [[34m2019-11-04 
> 16:27:37,298[0m] 
> [34mlocal_task_job.py:[0m105}} INFO[0m - Task exited with return code 0[0m
>  Also discussed in https://github.com/apache/airflow/pull/5489



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] akki opened a new pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator

akki opened a new pull request #6552: AIRFLOW-5850: Capture task logs in 
DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5850
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit test:
   `test_logging`
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-5901) Group tests for the tasks and variables command



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula updated AIRFLOW-5901:
---
Summary: Group tests for the tasks and variables command  (was: Group tests 
for the tasks command)

> Group tests for the tasks and variables command
> ---
>
> Key: AIRFLOW-5901
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5901
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6527: [AIRFLOW-5876][part of AIRFLOW-5697] Fetch all Dataflow jobs during searching

codecov-io edited a comment on issue #6527: [AIRFLOW-5876][part of 
AIRFLOW-5697] Fetch all Dataflow jobs during searching
URL: https://github.com/apache/airflow/pull/6527#issuecomment-551934161
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=h1) 
Report
   > Merging 
[#6527](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/56dd819427f1c6a2c907ad409951451f76f808bb?src=pr=desc)
 will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6527/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #6527  +/-   ##
   =
   - Coverage   84.08%   83.8%   -0.29% 
   =
 Files 638 639   +1 
 Lines   36858   36971 +113 
   =
   - Hits30992   30982  -10 
   - Misses   58665989 +123
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=)
 | `90.43% <100%> (-1.16%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `95.12% <0%> (-1.55%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.15% <0%> (-0.17%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/pagerduty\_hook.py](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3BhZ2VyZHV0eV9ob29rLnB5)
 | `81.57% <0%> (ø)` | |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/6527/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=footer). 
Last update 
[56dd819...2d36a03](https://codecov.io/gh/apache/airflow/pull/6527?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5901) Group tests for the tasks command

Kamil Bregula created AIRFLOW-5901:
--

 Summary: Group tests for the tasks command
 Key: AIRFLOW-5901
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5901
 Project: Apache Airflow
  Issue Type: Sub-task
  Components: cli
Affects Versions: 1.10.6
Reporter: Kamil Bregula






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] KevinYang21 commented on issue #6550: [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver

KevinYang21 commented on issue #6550: [AIRFLOW-5897] Allow setting -1 as pool 
slots value in webserver
URL: https://github.com/apache/airflow/pull/6550#issuecomment-552692114
 
 
   @pingzh @saguziel PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5900) heartbeat_callback for local_task_job can be more efficient



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971947#comment-16971947
 ] 

ASF GitHub Bot commented on AIRFLOW-5900:
-

houqp commented on pull request #6551: [AIRFLOW-5900] avoid unnecessary system 
calls in heartbeat_callback
URL: https://github.com/apache/airflow/pull/6551
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Optimize hearbeat callback, avoid system calls when they are not needed.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Existing tests already covered the change.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> heartbeat_callback for local_task_job can be more efficient
> ---
>
> Key: AIRFLOW-5900
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5900
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> local_task_job's heartbeat_callback function is issuing unnecessary system 
> calls when they are not needed. We can move those calls into the if block to 
> make it more efficient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] houqp opened a new pull request #6551: [AIRFLOW-5900] avoid unnecessary system calls in heartbeat_callback

houqp opened a new pull request #6551: [AIRFLOW-5900] avoid unnecessary system 
calls in heartbeat_callback
URL: https://github.com/apache/airflow/pull/6551
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Optimize hearbeat callback, avoid system calls when they are not needed.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Existing tests already covered the change.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5900) heartbeat_callback for local_task_job can be more efficient

QP Hou created AIRFLOW-5900:
---

 Summary: heartbeat_callback for local_task_job can be more 
efficient
 Key: AIRFLOW-5900
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5900
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


local_task_job's heartbeat_callback function is issuing unnecessary system 
calls when they are not needed. We can move those calls into the if block to 
make it more efficient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj edited a comment on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

mik-laj edited a comment on issue #6536: [AIRFLOW-5885] List of tests is 
generated by pre-commit not Docker build
URL: https://github.com/apache/airflow/pull/6536#issuecomment-552681674
 
 
   Is it not possible to generate this file at another time, e.g. when starting 
the breeze environment?  I am afraid that the existence of this file will cause 
many problems, because it will need to be updated forever. I would prefer that 
every change changes only what is necessary.  This makes it very difficult for 
beginners to add new functions because they may not have pre-commit installed. 
pre-commit is optional, but it is recommended to use it. The breeze environment 
is also optional, but it is recommended to use it. Changes to patches should 
also be possible for people who do not want to use the Breeze environment. 
Airflow is used by many companies and they may have their own forks, so 
additional files in the patch will hinder cherry-pick.
   
   BTW. Personally, I don't use automatic completion of test names, because I 
prefer to specify them by file name, and this is provided by Bash, so the 
necessity of completing a file that I don't use would cause a lot of emotion if 
it caused problems.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6549: [AIRFLOW-5880] Enforce unique task ids

mik-laj commented on a change in pull request #6549: [AIRFLOW-5880] Enforce 
unique task ids
URL: https://github.com/apache/airflow/pull/6549#discussion_r344973262
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -1190,13 +1189,8 @@ def add_task(self, task):
 task.end_date = min(task.end_date, self.end_date)
 
 if task.task_id in self.task_dict:
 
 Review comment:
   ```python
   dag =  DAG("test_dag", start_date=DEFAULT_DATE):
   dag.add_task(t1)
   dag.add_task(t1)
   ```
   Will this code still work?  In my opinion, re-adding the same task should 
not cause an exception.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6549: [AIRFLOW-5880] Enforce unique task ids

mik-laj commented on a change in pull request #6549: [AIRFLOW-5880] Enforce 
unique task ids
URL: https://github.com/apache/airflow/pull/6549#discussion_r344973262
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -1190,13 +1189,8 @@ def add_task(self, task):
 task.end_date = min(task.end_date, self.end_date)
 
 if task.task_id in self.task_dict:
 
 Review comment:
   ```
   dag =  DAG("test_dag", start_date=DEFAULT_DATE):
   dag.add_task(t1)
   dag.add_task(t1)
   ```
   Will this code still work?  In my opinion, re-adding the same task should 
not cause an exception.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5897) Allow setting -1 as pool slots value in webserver



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971936#comment-16971936
 ] 

ASF GitHub Bot commented on AIRFLOW-5897:
-

KevinYang21 commented on pull request #6550: [AIRFLOW-5897] Allow setting -1 as 
pool slots value in webserver
URL: https://github.com/apache/airflow/pull/6550
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5897) issues and references 
them in the PR title.
 - https://issues.apache.org/jira/browse/AIRFLOW-5897
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Given https://github.com/apache/airflow/pull/6520, we should also allow -1 
as pool slots value in the webserver.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Too minor a change to add unit test. Tested locally.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow setting -1 as pool slots value in webserver
> -
>
> Key: AIRFLOW-5897
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5897
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 2.0.0
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Major
>
> Given https://github.com/apache/airflow/pull/6520, we should also allow -1 as 
> pool slots value in the webserver.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] KevinYang21 opened a new pull request #6550: [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver

KevinYang21 opened a new pull request #6550: [AIRFLOW-5897] Allow setting -1 as 
pool slots value in webserver
URL: https://github.com/apache/airflow/pull/6550
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5897) issues and references 
them in the PR title.
 - https://issues.apache.org/jira/browse/AIRFLOW-5897
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Given https://github.com/apache/airflow/pull/6520, we should also allow -1 
as pool slots value in the webserver.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Too minor a change to add unit test. Tested locally.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

mik-laj commented on issue #6536: [AIRFLOW-5885] List of tests is generated by 
pre-commit not Docker build
URL: https://github.com/apache/airflow/pull/6536#issuecomment-552681674
 
 
   Is it not possible to generate this file at another time, e.g. when starting 
the breeze environment?  I am afraid that the existence of this file will cause 
many problems, because it will need to be updated forever. I would prefer that 
every change changes only what is necessary.  This makes it very difficult for 
beginners to add new functions because they may not have pre-commit installed. 
pre-commit is optional, but it is recommended to use it. The breeze environment 
is also optional, but it is recommended to use it. Changes to patches should 
also be possible for people who do not want to use the Breeze environment. 
Airflow is used by many companies and they may have their own forks, so 
additional files in the patch will hinder cherry-pick.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group tests for the Dags command

mik-laj commented on a change in pull request #6543: [AIRFLOW-5894][part of 
AIRFLOW-5893] Group tests for the Dags command
URL: https://github.com/apache/airflow/pull/6543#discussion_r344969914
 
 

 ##
 File path: tests/cli/test_cli.py
 ##
 @@ -272,13 +279,10 @@ def test_backfill(self, mock_run):
 '-s', DEFAULT_DATE.isoformat()]), dag=dag)
 
 mock_stdout.seek(0, 0)
-self.assertListEqual(
-[
-"Dry run of DAG example_bash_operator on 
{}\n".format(DEFAULT_DATE.isoformat()),
-"Task runme_0\n",
-],
-mock_stdout.readlines()
-)
+
+output = mock_stdout.read()
+self.assertIn("Dry run of DAG example_bash_operator on 
{}\n".format(DEFAULT_DATE.isoformat()), output)
+self.assertIn("Task runme_0\n".format(DEFAULT_DATE.isoformat()), 
output)
 
 Review comment:
   Airflow has a side effect that will be difficult to fix, so I changed this 
test a bit.
   The following message began to appear at the beginning of the output.
   ```
   [2019-11-12 00:15:08,234] {cli_action_loggers.py:67} DEBUG - Calling 
callbacks: []
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from tests/core.py

mik-laj commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from 
tests/core.py
URL: https://github.com/apache/airflow/pull/6544#issuecomment-552678321
 
 
   @potiuk  Every time I use pre-commit, it causes wrong file changes. I have 
to change them to make Travis happy. I do not know the reason for this problem, 
so I had to turn it off to be able to continue working on this project.
   For example:
   ```diff
   @@ -41,6 +41,8 @@ from dateutil.relativedelta import relativedelta
from numpy.testing import assert_array_almost_equal
from pendulum import utcnow
   
   +from tests.test_utils.config import conf_vars
   +
from airflow import DAG, configuration, exceptions, jobs, models, settings, 
utils
from airflow.bin import cli
from airflow.configuration import AirflowConfigException, conf, run_command
   @@ -62,7 +64,6 @@ from airflow.utils.dates import days_ago, infer_time_unit, 
round_time, scale_tim
from airflow.utils.state import State
from airflow.utils.timezone import datetime
from airflow.version import version
   -from tests.test_utils.config import conf_vars
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-5880) Enforce unique task ids



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik reassigned AIRFLOW-5880:
---

Assignee: Kaxil Naik

> Enforce unique task ids
> ---
>
> Key: AIRFLOW-5880
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5880
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: Kaxil Naik
>Priority: Major
>
> Currently, task with equal ids in a DAG simply overwrite each other and the 
> last assigned task is saved. This can lead to unexpected behaviour and an 
> exception should be raised.
> We've had a PendingDeprecationWarning on this for 4 years, time to get rid of 
> it: 
> [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5880) Enforce unique task ids



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971927#comment-16971927
 ] 

ASF GitHub Bot commented on AIRFLOW-5880:
-

kaxil commented on pull request #6549: [AIRFLOW-5880] Enforce unique task ids
URL: https://github.com/apache/airflow/pull/6549
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5880
   
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Currently, task with equal ids in a DAG simply overwrite each other and the 
last assigned task is saved. This can lead to unexpected behaviour and an 
exception should be raised.
   
   We've had a PendingDeprecationWarning on this for 4 years, time to get rid 
of it: 
https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enforce unique task ids
> ---
>
> Key: AIRFLOW-5880
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5880
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Priority: Major
>
> Currently, task with equal ids in a DAG simply overwrite each other and the 
> last assigned task is saved. This can lead to unexpected behaviour and an 
> exception should be raised.
> We've had a PendingDeprecationWarning on this for 4 years, time to get rid of 
> it: 
> [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #6549: [AIRFLOW-5880] Enforce unique task ids

kaxil opened a new pull request #6549: [AIRFLOW-5880] Enforce unique task ids
URL: https://github.com/apache/airflow/pull/6549
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5880
   
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Currently, task with equal ids in a DAG simply overwrite each other and the 
last assigned task is saved. This can lead to unexpected behaviour and an 
exception should be raised.
   
   We've had a PendingDeprecationWarning on this for 4 years, time to get rid 
of it: 
https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5899) dmypy still reports error



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971914#comment-16971914
 ] 

ASF GitHub Bot commented on AIRFLOW-5899:
-

houqp commented on pull request #6548: [AIRFLOW-5899] fix dmypy errors
URL: https://github.com/apache/airflow/pull/6548
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR fixes all errors reported by dmypy
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Should have been covered by mypy check
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> dmypy still reports error
> -
>
> Key: AIRFLOW-5899
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5899
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> dmypy is still reporting error, would be nice to get those fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] houqp opened a new pull request #6548: [AIRFLOW-5899] fix dmypy errors

houqp opened a new pull request #6548: [AIRFLOW-5899] fix dmypy errors
URL: https://github.com/apache/airflow/pull/6548
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR fixes all errors reported by dmypy
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Should have been covered by mypy check
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5899) dmypy still reports error

QP Hou created AIRFLOW-5899:
---

 Summary: dmypy still reports error
 Key: AIRFLOW-5899
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5899
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


dmypy is still reporting error, would be nice to get those fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5898) alembic command crashes due to typing import path collision



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971911#comment-16971911
 ] 

ASF GitHub Bot commented on AIRFLOW-5898:
-

houqp commented on pull request #6547: [AIRFLOW-5898] fix alembic crash due to 
typing import
URL: https://github.com/apache/airflow/pull/6547
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Running alemibc within airflow directory results in the following crash:
   
   ```
 File 
"/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", 
line 17, in 
   from typing import Generic, Callable, TypeVar, Tuple
   ImportError: cannot import name 'Generic' from 'typing'
   ```
   
   This is due to typing.py within airlfow directory collides with the builtin 
typing module when a command runs inside airflow directory. Renaming it to 
`custom_typing.py` fixes the issue.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   This should have been covered by mypy check.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> alembic command crashes due to typing import path collision
> ---
>
> Key: AIRFLOW-5898
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: database
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> Running alemibc within airflow directory results in the following crash:
>  
> {code:java}
>   File 
> "/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", 
> line 17, in 
> from typing import Generic, Callable, TypeVar, Tuple
> ImportError: cannot import name 'Generic' from 'typing'
> {code}
> This is due to typing.py within airlfow directory collides with the builtin 
> typing module when a command runs inside airflow directory. Renaming it to 
> `custom_typing.py` fixes the issue.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] houqp opened a new pull request #6547: [AIRFLOW-5898] fix alembic crash due to typing import

houqp opened a new pull request #6547: [AIRFLOW-5898] fix alembic crash due to 
typing import
URL: https://github.com/apache/airflow/pull/6547
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Running alemibc within airflow directory results in the following crash:
   
   ```
 File 
"/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", 
line 17, in 
   from typing import Generic, Callable, TypeVar, Tuple
   ImportError: cannot import name 'Generic' from 'typing'
   ```
   
   This is due to typing.py within airlfow directory collides with the builtin 
typing module when a command runs inside airflow directory. Renaming it to 
`custom_typing.py` fixes the issue.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   This should have been covered by mypy check.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5898) alembic command crashes due to typing import path collision

QP Hou created AIRFLOW-5898:
---

 Summary: alembic command crashes due to typing import path 
collision
 Key: AIRFLOW-5898
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
 Project: Apache Airflow
  Issue Type: New Feature
  Components: database
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Running alemibc within airflow directory results in the following crash:

 
{code:java}
  File 
"/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", 
line 17, in 
from typing import Generic, Callable, TypeVar, Tuple
ImportError: cannot import name 'Generic' from 'typing'
{code}
This is due to typing.py within airlfow directory collides with the builtin 
typing module when a command runs inside airflow directory. Renaming it to 
`custom_typing.py` fixes the issue.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-5897) Allow setting -1 as pool slots value in webserver

2019-11-11 Thread Kevin Yang (Jira)

Kevin Yang created AIRFLOW-5897:
---

 Summary: Allow setting -1 as pool slots value in webserver
 Key: AIRFLOW-5897
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5897
 Project: Apache Airflow
  Issue Type: Improvement
  Components: webserver
Affects Versions: 2.0.0
Reporter: Kevin Yang
Assignee: Kevin Yang


Given https://github.com/apache/airflow/pull/6520, we should also allow -1 as 
pool slots value in the webserver.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mattbowden-ookla opened a new pull request #6546: [AIRFLOW-XXX] Add render_template changes to UPDATING.md

mattbowden-ookla opened a new pull request #6546: [AIRFLOW-XXX] Add
render_template changes to UPDATING.md
URL: https://github.com/apache/airflow/pull/6546

Make sure you have checked _all_ steps below.

### Jira

- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [X] Here are some details about my PR, including screenshots of any UI
changes:

We recently encountered some errors in 1.10.6 that related to a specific
change to the `BaseOperator::render_template` function. This appears to be a
new change since 1.10.5:
- 1.10.5:
https://github.com/apache/airflow/blob/1.10.5/airflow/models/baseoperator.py#L667
- 1.10.6:
https://github.com/apache/airflow/blob/1.10.6/airflow/models/baseoperator.py#L699

We had upgraded from 1.10.4 to 1.10.6, and I was surprised when I first
looked at the code at both of those tags that it _appeared_ as though this
change was a part of 1.10.4 (even though our DAGs had been running fine in
1.10.4 but broke in 1.10.6).
Here's the code at the 1.10.4 git tag:
https://github.com/apache/airflow/blob/1.10.4/airflow/models/baseoperator.py#L699-L702
Here's the code at the 1.10.6 git tag:
https://github.com/apache/airflow/blob/1.10.6/airflow/models/baseoperator.py#L699

It would appear as though the 1.10.4 tag might be incorrect. It looks like
1.10.4b2 has the more accurate version of the code:
https://github.com/apache/airflow/blob/1.10.4b2/airflow/models/baseoperator.py#L660

I'm not sure how that might be resolved, but it's a separate problem from
this docs change.

I wasn't sure where to put this in the document, but it looked like it was
"most recent at the top" so I followed that. Additionally, should a new section
be made for 1.10.6 that's separate from `Airflow Master`? I can do that as part
of this PR too if it's desired.

### Tests

- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

This is just a docs change

### Commits

- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

potiuk commented on a change in pull request #6538: [AIRFLOW-5886] Selective 
copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]
URL: https://github.com/apache/airflow/pull/6538#discussion_r344932878
 
 

 ##
 File path: scripts/ci/local_ci_extract_tests.sh
 ##
 @@ -21,19 +21,25 @@ set -euxo pipefail
 
 MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1; pwd )"
 
-AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../.." || exit 1; pwd)
-export AIRFLOW_SOURCES
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
 
-gosu "${AIRFLOW_USER}" nosetests --collect-only --with-xunit 
--xunit-file="${HOME}/all_tests.xml"
+basic_sanity_checks
 
-gosu "${AIRFLOW_USER}" \
-python "${AIRFLOW_SOURCES}/tests/test_utils/get_all_tests.py" \
-"${HOME}/all_tests.xml" >"${HOME}/all_tests.txt"; \
+script_start
 
-echo ". ${HOME}/.bash_completion" >> "${HOME}/.bashrc"
+export PYTHON_VERSION=3.5
 
 Review comment:
   Yep. It's part of the earlier commit #6536  - i tried to make sure it is 
noticed in the comment (please review only last commit) but it's easy to misse. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

potiuk commented on a change in pull request #6538: [AIRFLOW-5886] Selective 
copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]
URL: https://github.com/apache/airflow/pull/6538#discussion_r344932878
 
 

 ##
 File path: scripts/ci/local_ci_extract_tests.sh
 ##
 @@ -21,19 +21,25 @@ set -euxo pipefail
 
 MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1; pwd )"
 
-AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../.." || exit 1; pwd)
-export AIRFLOW_SOURCES
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
 
-gosu "${AIRFLOW_USER}" nosetests --collect-only --with-xunit 
--xunit-file="${HOME}/all_tests.xml"
+basic_sanity_checks
 
-gosu "${AIRFLOW_USER}" \
-python "${AIRFLOW_SOURCES}/tests/test_utils/get_all_tests.py" \
-"${HOME}/all_tests.xml" >"${HOME}/all_tests.txt"; \
+script_start
 
-echo ". ${HOME}/.bash_completion" >> "${HOME}/.bashrc"
+export PYTHON_VERSION=3.5
 
 Review comment:
   Yep. It's part of the earlier commit #6536  - i tried to make sure it is 
noticed in the comment (please review only last commit) but it's easy to miss. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

kaxil commented on a change in pull request #6538: [AIRFLOW-5886] Selective 
copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]
URL: https://github.com/apache/airflow/pull/6538#discussion_r344929538
 
 

 ##
 File path: scripts/ci/local_ci_extract_tests.sh
 ##
 @@ -21,19 +21,25 @@ set -euxo pipefail
 
 MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1; pwd )"
 
-AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../.." || exit 1; pwd)
-export AIRFLOW_SOURCES
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
 
-gosu "${AIRFLOW_USER}" nosetests --collect-only --with-xunit 
--xunit-file="${HOME}/all_tests.xml"
+basic_sanity_checks
 
-gosu "${AIRFLOW_USER}" \
-python "${AIRFLOW_SOURCES}/tests/test_utils/get_all_tests.py" \
-"${HOME}/all_tests.xml" >"${HOME}/all_tests.txt"; \
+script_start
 
-echo ". ${HOME}/.bash_completion" >> "${HOME}/.bashrc"
+export PYTHON_VERSION=3.5
 
 Review comment:
   Can we add a comment here on why we have hard-coded Py 3.5 please?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #6541: [AIRFLOW-5892] BashOperator does not create temporary shell script

kaxil commented on issue #6541: [AIRFLOW-5892] BashOperator does not create 
temporary shell script
URL: https://github.com/apache/airflow/pull/6541#issuecomment-552634677
 
 
   It looks like a good improvement. Although, it would be better if we can add 
the problem that is solved by this PR in the JIRA Issue - 
https://issues.apache.org/jira/browse/AIRFLOW-5892 and the PR description too, 
please.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on issue #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on issue #6484: [AIRFLOW-5832] add pagerduty hook
URL: https://github.com/apache/airflow/pull/6484#issuecomment-552633369
 
 
   Thanks @kaxil and @pgagnon for the reviews!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-5832) pagerduty hook

2019-11-11 Thread ASF subversion and git services (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-5832.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> pagerduty hook
> --
>
> Key: AIRFLOW-5832
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5832
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
> Fix For: 2.0.0
>
>
> With pagerduty hook, users will be able to use pagerduty to notify critical 
> DAG run issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5832) pagerduty hook



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971862#comment-16971862
 ] 

ASF subversion and git services commented on AIRFLOW-5832:
--

Commit d25ef2b2abc3da9f5725a3c433bc035b970eec23 in airflow's branch 
refs/heads/master from Qingping Hou
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d25ef2b ]

[AIRFLOW-5832] Add pagerduty hook (#6484)



> pagerduty hook
> --
>
> Key: AIRFLOW-5832
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5832
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> With pagerduty hook, users will be able to use pagerduty to notify critical 
> DAG run issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil commented on issue #6484: [AIRFLOW-5832] add pagerduty hook

kaxil commented on issue #6484: [AIRFLOW-5832] add pagerduty hook
URL: https://github.com/apache/airflow/pull/6484#issuecomment-552631785
 
 
   Thanks @houqp 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5832) pagerduty hook



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971860#comment-16971860
 ] 

ASF GitHub Bot commented on AIRFLOW-5832:
-

kaxil commented on pull request #6484: [AIRFLOW-5832] add pagerduty hook
URL: https://github.com/apache/airflow/pull/6484
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> pagerduty hook
> --
>
> Key: AIRFLOW-5832
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5832
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> With pagerduty hook, users will be able to use pagerduty to notify critical 
> DAG run issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

kaxil commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344924689
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+self.token = conn.get_password()
 
 Review comment:
   No, that's fine :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #6484: [AIRFLOW-5832] add pagerduty hook

kaxil merged pull request #6484: [AIRFLOW-5832] add pagerduty hook
URL: https://github.com/apache/airflow/pull/6484
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dstandish commented on a change in pull request #6440: [AIRFLOW-5768] GCP cloud sql don't store ephemeral connection in db

dstandish commented on a change in pull request #6440: [AIRFLOW-5768] GCP cloud 
sql don't store ephemeral connection in db
URL: https://github.com/apache/airflow/pull/6440#discussion_r344923202
 
 

 ##
 File path: tests/gcp/hooks/test_cloud_sql.py
 ##
 @@ -1213,235 +1173,144 @@ def test_get_sqlproxy_runner(self):
 )
 self.assertEqual(sqlproxy_runner.instance_specification, instance_spec)
 
-@mock.patch("airflow.hooks.base_hook.BaseHook.get_connections")
-def test_hook_with_not_too_long_unix_socket_path(self, get_connections):
+
@mock.patch("airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook.get_connection")
 
 Review comment:
   are you asking me to move objects in this PR, in addition to the other 
changes?  shouldn't any object moving be done in separate PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dstandish commented on issue #6440: [AIRFLOW-5768] GCP cloud sql don't store ephemeral connection in db

dstandish commented on issue #6440: [AIRFLOW-5768] GCP cloud sql don't store 
ephemeral connection in db
URL: https://github.com/apache/airflow/pull/6440#issuecomment-552629484
 
 
   thanks @potiuk 
   
   i have rebased


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344920107
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+self.token = conn.get_password()
 
 Review comment:
   I started with `password`, but noticed it's just a wrapper around 
get_password and set_password, so I figured it's more efficient(less func call) 
and readable(less magic) to use `get_password` here instead since we only need 
to read the content.
   
   Let me know if you still think the `password` wrapper is more readable 
though, i am more than happy to change it :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

kaxil commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344902795
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+self.token = conn.get_password()
 
 Review comment:
   ```suggestion
   self.token = conn.password
   ```
   
   should work too


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6545: [AIRFLOW-5896] Move email stuff from tests/core.py

potiuk commented on issue #6545: [AIRFLOW-5896] Move email stuff from 
tests/core.py
URL: https://github.com/apache/airflow/pull/6545#issuecomment-552597676
 
 
   Travis is sad here as well :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from tests/core.py

potiuk commented on issue #6544: [AIRFLOW-5895] Move HDFS stuff from 
tests/core.py
URL: https://github.com/apache/airflow/pull/6544#issuecomment-552597512
 
 
   Travis is sad here as well. I recommend to run pre-commit hooks every time 
@mik-laj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group tests for the Dags command

potiuk commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group 
tests for the Dags command
URL: https://github.com/apache/airflow/pull/6543#issuecomment-552597215
 
 
   Travis is sad!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344890371
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+
+self.token = getattr(conn, 'password', None)
+
+routing_key = conn.extra_dejson.get("routing_key")
+if routing_key:
+self.routing_key = routing_key
+
+if token is not None:  # token takes higher priority
+self.token = token
+
+if self.token is None:
+raise AirflowException(
+'Cannot get token: No valid api token nor pagerduty_conn_id 
supplied.')
+
 
 Review comment:
   Making this change will alter the actual logic. Because it will throw an 
error when token is `None` when there is a valid password set in connection. We 
want token to override password from connection, but it's not required.
   
   This case has been covered in test cases as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344890371
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+
+self.token = getattr(conn, 'password', None)
+
+routing_key = conn.extra_dejson.get("routing_key")
+if routing_key:
+self.routing_key = routing_key
+
+if token is not None:  # token takes higher priority
+self.token = token
+
+if self.token is None:
+raise AirflowException(
+'Cannot get token: No valid api token nor pagerduty_conn_id 
supplied.')
+
 
 Review comment:
   Making this change will alter the actual logic. Because it will throw an 
error when token is `None` when there is a valid password set in connection. We 
want token to override password from connection if needed, but it's not 
required.
   
   This case has been covered in test cases as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344889535
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+
+self.token = getattr(conn, 'password', None)
+
+routing_key = conn.extra_dejson.get("routing_key")
 
 Review comment:
   `extra_dejson` returns a dictionary, so `get("foo")` is the same as 
`get("goo", None)`. I have added a test case to make sure missing routing_key 
extra config won't cause any crash.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty hook

houqp commented on a change in pull request #6484: [AIRFLOW-5832] add pagerduty 
hook
URL: https://github.com/apache/airflow/pull/6484#discussion_r344889060
 
 

 ##
 File path: airflow/contrib/hooks/pagerduty_hook.py
 ##
 @@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Hook for creating Pagerduty incidents."""
+from typing import Any, Dict, List, Optional
+
+import pypd
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class PagerdutyHook(BaseHook):
+"""
+Takes both Pagerduty API token directly and connection that has Pagerduty 
API token.
+
+If both supplied, Pagerduty API token will be used.
+
+:param token: Pagerduty API token
+:param pagerduty_conn_id: connection that has Pagerduty API token in the 
password field
+"""
+
+def __init__(self, token: Optional[str] = None, pagerduty_conn_id: 
Optional[str] = None) -> None:
+self.routing_key = None
+
+if pagerduty_conn_id is not None:
+conn = self.get_connection(pagerduty_conn_id)
+
+self.token = getattr(conn, 'password', None)
 
 Review comment:
   good catch, I copied it from the slack hook without digging into password 
implementation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6440: [AIRFLOW-5768] GCP cloud sql don't store ephemeral connection in db

potiuk commented on a change in pull request #6440: [AIRFLOW-5768] GCP cloud 
sql don't store ephemeral connection in db
URL: https://github.com/apache/airflow/pull/6440#discussion_r344885576
 
 

 ##
 File path: tests/gcp/hooks/test_cloud_sql.py
 ##
 @@ -1213,235 +1173,144 @@ def test_get_sqlproxy_runner(self):
 )
 self.assertEqual(sqlproxy_runner.instance_specification, instance_spec)
 
-@mock.patch("airflow.hooks.base_hook.BaseHook.get_connections")
-def test_hook_with_not_too_long_unix_socket_path(self, get_connections):
+
@mock.patch("airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook.get_connection")
 
 Review comment:
   Should we move it to "airflow.google.cloud.hooks" ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6541: [AIRFLOW-5892] BashOperator does not create temporary shell script

codecov-io edited a comment on issue #6541: [AIRFLOW-5892] BashOperator does 
not create temporary shell script
URL: https://github.com/apache/airflow/pull/6541#issuecomment-552592718
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=h1) 
Report
   > Merging 
[#6541](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/3fa64ea7a9476de4810f3b95f170074a2b422206?src=pr=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `77.77%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6541/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6541  +/-   ##
   ==
   - Coverage   83.83%   83.78%   -0.06% 
   ==
 Files 638  638  
 Lines   3686136857   -4 
   ==
   - Hits3090230879  -23 
   - Misses   5959 5978  +19
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmFzaF9vcGVyYXRvci5weQ==)
 | `90.56% <77.77%> (-0.82%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `96.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5)
 | `95.78% <0%> (+0.04%)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.79% <0%> (+0.5%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.52% <0%> (+0.84%)` | :arrow_up: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=footer). 
Last update 
[3fa64ea...4a7db18](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6541: [AIRFLOW-5892] BashOperator does not create temporary shell script

codecov-io commented on issue #6541: [AIRFLOW-5892] BashOperator does not 
create temporary shell script
URL: https://github.com/apache/airflow/pull/6541#issuecomment-552592718
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=h1) 
Report
   > Merging 
[#6541](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/3fa64ea7a9476de4810f3b95f170074a2b422206?src=pr=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `77.77%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6541/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6541  +/-   ##
   ==
   - Coverage   83.83%   83.78%   -0.06% 
   ==
 Files 638  638  
 Lines   3686136857   -4 
   ==
   - Hits3090230879  -23 
   - Misses   5959 5978  +19
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmFzaF9vcGVyYXRvci5weQ==)
 | `90.56% <77.77%> (-0.82%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `96.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5)
 | `95.78% <0%> (+0.04%)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.79% <0%> (+0.5%)` | :arrow_up: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.52% <0%> (+0.84%)` | :arrow_up: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/6541/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=footer). 
Last update 
[3fa64ea...4a7db18](https://codecov.io/gh/apache/airflow/pull/6541?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6447: [AIRFLOW-5792] Straighten out the migrations

potiuk commented on issue #6447: [AIRFLOW-5792] Straighten out the migrations
URL: https://github.com/apache/airflow/pull/6447#issuecomment-552590973
 
 
   Not sure what was the last agreement we had @fokko ? was I supposed to do 
this or did we agree you do it :) ? Just got some busy week last week :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5890) Many dag task status is skiped



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971822#comment-16971822
 ] 

Kaxil Naik commented on AIRFLOW-5890:
-

Need more information !

Environment details, airflow configuration (airflow.cfg) and your DAG. 

> Many dag task status is skiped
> --
>
> Key: AIRFLOW-5890
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5890
> Project: Apache Airflow
>  Issue Type: Task
>  Components: DAG
>Affects Versions: 1.10.5
>Reporter: yangming
>Priority: Major
> Attachments: wx2019-132...@2x.png, wx2019-132...@2x.png
>
>
> There ars some dag tasks are skiped in our production environment and I have 
> no way to solve it. !wx2019-132...@2x.png!!wx2019-132...@2x.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (AIRFLOW-5891) Delete pervasively existed DAGs



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik closed AIRFLOW-5891.
---
Resolution: Implemented

Fixed in https://issues.apache.org/jira/browse/AIRFLOW-5481

> Delete pervasively existed DAGs 
> 
>
> Key: AIRFLOW-5891
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5891
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: DAG
>Affects Versions: 1.9.0
>Reporter: London Su
>Priority: Major
> Attachments: image-2019-11-11-15-23-29-532.png
>
>
> After delete or rename old DAGs, apart from creating new DAGs, users should 
> be able to delete the old DAGs.
>  
> In the screenshot below, *processors.airbnb_search_listings* is an old DAG, 
> it was renamed to *processors.airbnb_search_listings_v0.1*. It would be good 
> if the old DAG can be deleted.
> !image-2019-11-11-15-23-29-532.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group tests for the Dags command

potiuk commented on issue #6543: [AIRFLOW-5894][part of AIRFLOW-5893] Group 
tests for the Dags command
URL: https://github.com/apache/airflow/pull/6543#issuecomment-552568462
 
 
   Approved. Travis is sad however.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6542: AIRFLOW-5854: Add support for `tty` parameter in Docker related operators

2019-11-11 Thread ASF subversion and git services (Jira)

potiuk commented on issue #6542: AIRFLOW-5854: Add support for `tty` parameter 
in Docker related operators
URL: https://github.com/apache/airflow/pull/6542#issuecomment-552566397
 
 
   Also cherry-picked it to 1.10.7


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5854) Docker logs of Python processes are not visible



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971790#comment-16971790
 ] 

ASF subversion and git services commented on AIRFLOW-5854:
--

Commit 10534a1044d8660dcaa86695fdde1ffeaa8829f7 in airflow's branch 
refs/heads/v1-10-test from Akshesh Doshi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=10534a1 ]

AIRFLOW-5854: Add support for `tty` parameter in Docker related operators 
(#6542)

 (cherry-picked from commit 7ab9e957d3ade7af607b597592b624c9e4951b73)


> Docker logs of Python processes are not visible
> ---
>
> Key: AIRFLOW-5854
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5854
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: logging, operators
>Affects Versions: 2.0.0
>Reporter: Akshesh Doshi
>Assignee: Akshesh Doshi
>Priority: Major
>  Labels: docker, orchestration, swarm
> Fix For: 2.0.0
>
>
> Reason - [https://stackoverflow.com/a/33027467/3061686]
> Fix - Add support for `tty` 
> ([https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec])
>  parameter in Docker Swarm operator (and set to True, by default).
>  
> This is also required to fix 
> https://issues.apache.org/jira/browse/AIRFLOW-5850.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5870) Optimise pool query and allow infinite pool for tasks

2019-11-11 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971787#comment-16971787
 ] 

ASF subversion and git services commented on AIRFLOW-5870:
--

Commit 0573f064b016f4e4f8494d8e061f571428096b70 in airflow's branch 
refs/heads/v1-10-test from Alex Guziel
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0573f06 ]

[AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)

  (cherry-picked from commit 4a12041ac06a038e0af68ce3e19e46200d2f6376)


> Optimise pool query and allow infinite pool for tasks
> -
>
> Key: AIRFLOW-5870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5870
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Alex Guziel
>Assignee: Alex Guziel
>Priority: Major
> Fix For: 1.10.7
>
>
> Pools do not allow infinite sized pools. Infinite sized pools can make 
> queries much cheaper 
>  
> The count() query in Pool query for number of tasks running was slow and this 
> change fixes it (as described here): 
> [https://stackoverflow.com/questions/12941416/how-to-count-rows-with-select-count-with-sqlalchemy]
> Also here: 
> [https://stackoverflow.com/questions/14754994/why-is-sqlalchemy-count-much-slower-than-the-raw-query]
>   
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-11 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971788#comment-16971788
 ] 

ASF subversion and git services commented on AIRFLOW-5888:
--

Commit 14d08fc46cd3a2735c3c5060d6d1a865cc1306f6 in airflow's branch 
refs/heads/v1-10-test from Matthew Bowden
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=14d08fc ]

[AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)

  (cherry-picked from commit b401759322d3e315fb23ace6c4ac996f6d1020eb)


> Use psycopg2-binary package
> ---
>
> Key: AIRFLOW-5888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5888
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: dependencies
>Affects Versions: 1.10.4, 1.10.5, 1.10.6
>Reporter: Matthew Bowden
>Assignee: Matthew Bowden
>Priority: Major
> Fix For: 1.10.7
>
>
> This requirement was pinned as of PR 5036, as related to a breaking change 
> with respect to the package namespace. This PR modifies which package to 
> refer to - prior to 2.8 the package was a binary wheel, but with 2.8+ the 
> appropriate installation package is {{psycopg2-binary}}
> h3. Explanation:
> [Full explanation 
> here|http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/]
> For most of psycopg2's history, they had distributed pre-compiled binaries 
> (or "wheels") which would get installed when you did {{pip install psycopg2}} 
> (rather than compiling the source code upon installation). At some point 
> psycopg2 found that there were some segfault issues that were appearing in 
> the pre-compiled versions. Granted, this could be fixed if you installed by 
> explicitly saying you wanted to compile from source rather than retrieve the 
> binary version, but the psycopg2 devs felt that this would be better 
> addressed by publishing a separate package that was the pre-compiled binary, 
> while keeping the original package as compile-from-source only. The upshot of 
> this is that everyone who was used to installing {{psycopg2}} and getting the 
> pre-compiled binaries was now having to compile the package on install unless 
> they moved over to {{psycopg2-binary}} instead. The packages are effectively 
> the same, only the pre-compiled versions are usually more optimized, run 
> quicker on the systems they're build for, and don't require compilation 
> dependencies to install. So, in accordance with the warning provided by 
> psycopg2 ({{UserWarning: The psycopg2 wheel package will be renamed from 
> release 2.8; in order to keep installing from binary please use "pip install 
> psycopg2-binary" instead.}}), I've swapping this over to psycopg2-binary.
> I've also confirmed that {{psycopg2-binary}} is available from {{2.7.4}} 
> onwards:
> $ pip install psycopg2-binary== Collecting psycopg2-binary== ERROR: Could not 
> find a version that satisfies the requirement psycopg2-binary== (from 
> versions: 2.7.4, 2.7.5, 2.7.6, 2.7.6.1, 2.7.7, 2.8, 2.8.1, 2.8.2, 2.8.3, 
> 2.8.4) ERROR: No matching distribution found {color:#d73a49}for{color} 
> psycopg2-binary==



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5869) Creating DagRuns fails for Deserialized tasks with no start_date

2019-11-11 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971789#comment-16971789
 ] 

ASF subversion and git services commented on AIRFLOW-5869:
--

Commit d3899dc2acbb031185de72cafbd1ea0057bb38fd in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d3899dc ]

[AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)


(cherry picked from commit 3fa64ea7a9476de4810f3b95f170074a2b422206)


> Creating DagRuns fails for Deserialized tasks with no start_date
> 
>
> Key: AIRFLOW-5869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5869
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.7
>
>
> Deserialized operators do not always have start_date set. 
> That, for instance, breaks triggering dags.
> See the code from DAG.create_dagrun():
> {code:python}
> run = DagRun(...)
> session.add(run)
> session.commit()
> run.dag = self
> run.verify_integrity(session=session) # this validation fails because 
> run assumes that all operators have start_date set
> run.refresh_from_db()
> {code}
> One of the optimisation 
> (https://github.com/coufon/airflow/commit/b5ee858f44f55818c589cf2c8bf3866fa5d50e30)
>  we did as part of DAG Serialization was to not store dates in tasks if they 
> have a matching date (start_date or end_date) in DAG. Unfortunately, when 
> triggering DAG containing such tasks, it fails on DagRun.run.verify_integrity.
> The fix is to add the start_date when deserializing the operator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5854) Docker logs of Python processes are not visible

2019-11-11 Thread Jarek Potiuk (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-5854:
--
Fix Version/s: (was: 2.0.0)
   1.10.7

> Docker logs of Python processes are not visible
> ---
>
> Key: AIRFLOW-5854
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5854
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: logging, operators
>Affects Versions: 2.0.0
>Reporter: Akshesh Doshi
>Assignee: Akshesh Doshi
>Priority: Major
>  Labels: docker, orchestration, swarm
> Fix For: 1.10.7
>
>
> Reason - [https://stackoverflow.com/a/33027467/3061686]
> Fix - Add support for `tty` 
> ([https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec])
>  parameter in Docker Swarm operator (and set to True, by default).
>  
> This is also required to fix 
> https://issues.apache.org/jira/browse/AIRFLOW-5850.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5854) Docker logs of Python processes are not visible

2019-11-11 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971775#comment-16971775
 ] 

ASF subversion and git services commented on AIRFLOW-5854:
--

Commit 7ab9e957d3ade7af607b597592b624c9e4951b73 in airflow's branch 
refs/heads/master from Akshesh Doshi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7ab9e95 ]

AIRFLOW-5854: Add support for `tty` parameter in Docker related operators 
(#6542)



> Docker logs of Python processes are not visible
> ---
>
> Key: AIRFLOW-5854
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5854
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: logging, operators
>Affects Versions: 2.0.0
>Reporter: Akshesh Doshi
>Assignee: Akshesh Doshi
>Priority: Major
>  Labels: docker, orchestration, swarm
> Fix For: 2.0.0
>
>
> Reason - [https://stackoverflow.com/a/33027467/3061686]
> Fix - Add support for `tty` 
> ([https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec])
>  parameter in Docker Swarm operator (and set to True, by default).
>  
> This is also required to fix 
> https://issues.apache.org/jira/browse/AIRFLOW-5850.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6542: AIRFLOW-5854: Add support for `tty` parameter in Docker related operators

potiuk merged pull request #6542: AIRFLOW-5854: Add support for `tty` parameter 
in Docker related operators
URL: https://github.com/apache/airflow/pull/6542
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5854) Docker logs of Python processes are not visible