date:20200218

MichaelChirico opened a new pull request #7463: [AIRFLOW-] clarify dag_id 
parameter
URL: https://github.com/apache/airflow/pull/7463
 
 
   Re-filing #7462 here:
   
   I spent a while chasing this just now, wondering whether `:` is valid before 
finding `validate_key()` in helpers.
   
   I think it's helpful to surface this here directly.
   
   Trivial change so eschewing the other formality required in the PR template


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] commented on issue #6501: [AIRFLOW-5831] Prod image support. Depends on [AIRFLOW-5704] [AIRFLOW-5842] [AIRFLOW-5828]

stale[bot] commented on issue #6501: [AIRFLOW-5831] Prod image support. Depends 
on [AIRFLOW-5704] [AIRFLOW-5842] [AIRFLOW-5828]
URL: https://github.com/apache/airflow/pull/6501#issuecomment-588084237
 
 
   This issue has been automatically marked as stale because it has not had 
recent activity. It will be closed if no further activity occurs. Thank you for 
your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] MichaelChirico closed pull request #7462: [AIRFLOW-XXXX] clarify dag_id parameter

MichaelChirico closed pull request #7462: [AIRFLOW-] clarify dag_id 
parameter
URL: https://github.com/apache/airflow/pull/7462
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] MichaelChirico commented on issue #7462: [AIRFLOW-XXXX] clarify dag_id parameter

MichaelChirico commented on issue #7462: [AIRFLOW-] clarify dag_id parameter
URL: https://github.com/apache/airflow/pull/7462#issuecomment-588083940
 
 
   let's try again


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on issue #7462: [AIRFLOW-XXXX] clarify dag_id parameter

zhongjiajie commented on issue #7462: [AIRFLOW-] clarify dag_id parameter
URL: https://github.com/apache/airflow/pull/7462#issuecomment-588071249
 
 
   @MichaelChirico  use `git push -f` to restart it. maybe?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie edited a comment on issue #7462: [AIRFLOW-XXXX] clarify dag_id parameter

zhongjiajie edited a comment on issue #7462: [AIRFLOW-] clarify dag_id 
parameter
URL: https://github.com/apache/airflow/pull/7462#issuecomment-588071249
 
 
   @MichaelChirico use `git push -f` to restart it. maybe?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] MichaelChirico commented on issue #7462: [AIRFLOW-XXXX] clarify dag_id parameter

MichaelChirico commented on issue #7462: [AIRFLOW-] clarify dag_id parameter
URL: https://github.com/apache/airflow/pull/7462#issuecomment-588065081
 
 
   Is there a way to re-trigger the Title Validator without adding commits?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7462: [Documentation] clarify dag_id parameter

boring-cyborg[bot] commented on issue #7462: [Documentation] clarify dag_id
parameter
URL: https://github.com/apache/airflow/pull/7462#issuecomment-588064513

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for
testing locally, it’s a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] MichaelChirico opened a new pull request #7462: [Documentation] clarify dag_id parameter

MichaelChirico opened a new pull request #7462: [Documentation] clarify dag_id 
parameter
URL: https://github.com/apache/airflow/pull/7462
 
 
   I spent a while chasing this just now, wondering whether `:` is valid before 
finding `validate_key()` in helpers.
   
   I think it's helpful to surface this here directly.
   
   Trivial change so eschewing the other formality required in the PR template


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codenamelxl commented on issue #7374: [AIRFLOW-6399] Serialization: DAG access_control field should be deco…

codenamelxl commented on issue #7374: [AIRFLOW-6399] Serialization: DAG 
access_control field should be deco…
URL: https://github.com/apache/airflow/pull/7374#issuecomment-588056628
 
 
   @kaxil Test run OK with Postgres. I'm not sure what wrong with the test for 
MySQL and SQLite.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7457: [AIRFLOW-6837] limit description len of dag in home

codecov-io commented on issue #7457: [AIRFLOW-6837] limit description len of 
dag in home
URL: https://github.com/apache/airflow/pull/7457#issuecomment-588048473
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=h1) 
Report
   > Merging 
[#7457](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/65dd28eb77d996ec8306c67d5ce1ccee2c14cc9d?src=pr=desc)
 will **decrease** coverage by `0.17%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7457/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7457  +/-   ##
   ==
   - Coverage   86.62%   86.45%   -0.18% 
   ==
 Files 878  878  
 Lines   4129241292  
   ==
   - Hits3577035697  -73 
   - Misses   5522 5595  +73
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0%> (-64.16%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55% <0%> (-45%)` | :arrow_down: |
   | 
[airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==)
 | `76.02% <0%> (-1.54%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `90.9% <0%> (-0.83%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.8% <0%> (-0.29%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7457/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `89.51% <0%> (-0.15%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=footer). 
Last update 
[65dd28e...089c99d](https://codecov.io/gh/apache/airflow/pull/7457?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7461: [AIRFLOW-XXXX] Update write-logs.rst, add instructions for logging to localstack S3

boring-cyborg[bot] commented on issue #7461: [AIRFLOW-] Update
write-logs.rst, add instructions for logging to localstack S3
URL: https://github.com/apache/airflow/pull/7461#issuecomment-588037518

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] redSlug opened a new pull request #7461: [AIRFLOW-XXXX] Update write-logs.rst, add instructions for logging to localstack S3

redSlug opened a new pull request #7461: [AIRFLOW-] Update write-logs.rst, 
add instructions for logging to localstack S3
URL: https://github.com/apache/airflow/pull/7461
 
 
   ---
   Issue link:  Document only change, no JIRA issue
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] simis2626 commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…

simis2626 commented on issue #7395: [AIRFLOW-5629] Implement K8s 
priorityClassName in KubernetesPo…
URL: https://github.com/apache/airflow/pull/7395#issuecomment-588032431
 
 
   Hi @dimberman 
   Appreciate your recent return to this work, just pinging you incase it's 
slipped by.
   Thanks in advance, appreciate your support & time.
   Simon


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

mik-laj commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real 
subcommands for Breeze
URL: https://github.com/apache/airflow/pull/7458#issuecomment-588015135
 
 
   I updated autocompllete. This should also solve the problems with getopt.
   
   > you can do ./breeeze  and it will do something instead of complaining.
   
   Thanks. I added missing check. 
   
   I still have to do rebase, but I will do it tomorrow.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands 
for Breeze
URL: https://github.com/apache/airflow/pull/7458#issuecomment-588009246
 
 
   Even if you provide --stop, get opt will suggest that it is ambigius and 
will tell you that --stop-environment is one of possible options.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands 
for Breeze
URL: https://github.com/apache/airflow/pull/7458#issuecomment-588009056
 
 
   Sure: getopt verifies switches provided and will complain if the switch is 
wrong. Try --nnn . with the command specified you get no verification you can 
do ./breeeze  and it will do something instead of complaining. If you 
specify --stop-environment as switch, it will pass over getopt but 
--stop-environments will fail in get_opt. And the list of options (available in 
breeze-complete) is not updated. The same list is used for autocomplete


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable

codecov-io edited a comment on issue #7459: [AIRFLOW-6820] fixup! fix unbound 
variable
URL: https://github.com/apache/airflow/pull/7459#issuecomment-588007834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=h1) 
Report
   > Merging 
[#7459](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/24f9f11775bf7b38cced9955f6c70705ad56f61c?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7459/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#7459   +/-   ##
   ===
 Coverage   86.62%   86.62%   
   ===
 Files 878  878   
 Lines   4129241292   
   ===
 Hits3577035770   
 Misses   5522 5522
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=footer). 
Last update 
[24f9f11...a54cf8b](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable

codecov-io commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable
URL: https://github.com/apache/airflow/pull/7459#issuecomment-588007834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=h1) 
Report
   > Merging 
[#7459](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/24f9f11775bf7b38cced9955f6c70705ad56f61c?src=pr=desc)
 will **decrease** coverage by `0.18%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7459/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7459  +/-   ##
   ==
   - Coverage   86.62%   86.44%   -0.19% 
   ==
 Files 878  878  
 Lines   4129241292  
   ==
   - Hits3577035694  -76 
   - Misses   5522 5598  +76
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0%> (-64.16%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55% <0%> (-45%)` | :arrow_down: |
   | 
[airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==)
 | `76.02% <0%> (-1.54%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `90.96% <0%> (-1.13%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `90.9% <0%> (-0.83%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7459/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `89.51% <0%> (-0.15%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=footer). 
Last update 
[24f9f11...a54cf8b](https://codecov.io/gh/apache/airflow/pull/7459?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions

2020-02-18 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039654#comment-17039654
 ] 

ASF GitHub Bot commented on AIRFLOW-6820:
-

potiuk commented on pull request #7459: [AIRFLOW-6820] fixup! fix unbound 
variable
URL: https://github.com/apache/airflow/pull/7459
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 1.10.10
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039655#comment-17039655
 ] 

ASF subversion and git services commented on AIRFLOW-6820:
--

Commit d2ab8b8f5f921361dc6f28e21c6e588693c0e81d in airflow's branch 
refs/heads/master from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d2ab8b8 ]

[AIRFLOW-6820] fixup! fix unbound variable (#7459)



> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 1.10.10
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7459: [AIRFLOW-6820] fixup! fix unbound variable

potiuk merged pull request #7459: [AIRFLOW-6820] fixup! fix unbound variable
URL: https://github.com/apache/airflow/pull/7459
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6839) Even more mypy speedup



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039653#comment-17039653
 ] 

ASF GitHub Bot commented on AIRFLOW-6839:
-

potiuk commented on pull request #7460: [AIRFLOW-6839] even more mypy speed 
improvements
URL: https://github.com/apache/airflow/pull/7460
 
 
   Require_serial:true is better choice than pass_filename: false as it can
   speed-up mypy for single file changes.
   
   Significant gains can be achieved for single file changes and no cache for 
all
   other files. This is majority of cases for our users who have pre-commits
   installed as hooks because most people change only few files and never run
   check with --all-files
   
   When just one file is changed and no cache is built, the difference is 
drastic:
   
   require_serial: true = 4s
   
   pass_filenames: false =  13s
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Even more mypy speedup
> --
>
> Key: AIRFLOW-6839
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6839
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: pre-commit
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
>
> require_serial:true is better choice than pass_filename: false as it can 
> speed-up mypy for single file changes.
> Significant gains can be achieved for single file changes and no cache for 
> all other files. This is majority of cases for our users who have pre-commits 
> installed as hooks because most people change only few files and never run 
> check with --all-files 
> When just one file is changed and no cache is built, the difference is 
> drastic:
> require_serial: true (4s) 
>  
> {code:java}
> Run 
> mypy.PassedRun
>  mypy.Passed
> real 0m4,080s user 0m0,283s sys 0m0,095s
> {code}
> pass_filenames: false: (13s)
> {code:java}
> time pre-commit run mypy time pre-commit run mypyRun 
> mypy.Passed
> real 0m13,756suser 0m0,281ssys 0m0,099s{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable

potiuk commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable
URL: https://github.com/apache/airflow/pull/7459#issuecomment-588002347
 
 
   I'll merge this now - this is a fixup to just merged commit with one 
variable set - to prevent people to get it with breeze (breeze was failing).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk opened a new pull request #7460: [AIRFLOW-6839] even more mypy speed improvements

potiuk opened a new pull request #7460: [AIRFLOW-6839] even more mypy speed 
improvements
URL: https://github.com/apache/airflow/pull/7460
 
 
   Require_serial:true is better choice than pass_filename: false as it can
   speed-up mypy for single file changes.
   
   Significant gains can be achieved for single file changes and no cache for 
all
   other files. This is majority of cases for our users who have pre-commits
   installed as hooks because most people change only few files and never run
   check with --all-files
   
   When just one file is changed and no cache is built, the difference is 
drastic:
   
   require_serial: true = 4s
   
   pass_filenames: false =  13s
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6839) Even more mypy speedup

Jarek Potiuk created AIRFLOW-6839:
-

 Summary: Even more mypy speedup
 Key: AIRFLOW-6839
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6839
 Project: Apache Airflow
  Issue Type: Improvement
  Components: pre-commit
Affects Versions: 1.10.9, 2.0.0
Reporter: Jarek Potiuk


require_serial:true is better choice than pass_filename: false as it can 
speed-up mypy for single file changes.

Significant gains can be achieved for single file changes and no cache for all 
other files. This is majority of cases for our users who have pre-commits 
installed as hooks because most people change only few files and never run 
check with --all-files 

When just one file is changed and no cache is built, the difference is drastic:

require_serial: true (4s) 

 
{code:java}
Run 
mypy.PassedRun 
mypy.Passed
real 0m4,080s user 0m0,283s sys 0m0,095s
{code}
pass_filenames: false: (13s)
{code:java}
time pre-commit run mypy time pre-commit run mypyRun 
mypy.Passed
real 0m13,756suser 0m0,281ssys 0m0,099s{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6818) Prevent Docker cache-busting on when editing www templates



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6818:
--
Affects Version/s: (was: 2.0.0)
   1.10.10

> Prevent Docker cache-busting on when editing www templates
> --
>
> Key: AIRFLOW-6818
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6818
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 1.10.10
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.10
>
>
> Due to permission differences on files/folders between Travis and docker hub 
> auto build we are needlessly busting the docker layer cache -> builds are a 
> bit slower than they need to be



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6823) Breeze broken on current master on Linux



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6823:
--
Affects Version/s: (was: 2.0.0)

> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 1.10.10
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6820) Split breeze into functions



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6820:
--
Affects Version/s: (was: 1.10.9)
   1.10.10

> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.10
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6820) Split breeze into functions



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6820:
--
Affects Version/s: (was: 2.0.0)

> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 1.10.10
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6816) Separate breeze's variable setup into functions

2020-02-18 Thread ASF subversion and git services (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6816.
---
Fix Version/s: 1.10.10
   Resolution: Fixed

> Separate breeze's variable setup into functions
> ---
>
> Key: AIRFLOW-6816
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6816
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.10
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6823) Breeze broken on current master on Linux



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039637#comment-17039637
 ] 

ASF subversion and git services commented on AIRFLOW-6823:
--

Commit dbe0bbde74ebadb271b2f80dc0508090b69d88d1 in airflow's branch 
refs/heads/v1-10-test from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dbe0bbd ]

[AIRFLOW-6823] Fix root owned files automatically in Breeze (#7449)


(cherry picked from commit 03fb50aaa4bd7b00968c300781039cebf91043f2)


> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 2.0.0
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6823) Breeze broken on current master on Linux



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6823:
--
Affects Version/s: 1.10.10

> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.10
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions

2020-02-18 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039640#comment-17039640
 ] 

ASF GitHub Bot commented on AIRFLOW-6820:
-

potiuk commented on pull request #7459: [AIRFLOW-6820] fixup! fix unbound 
variable
URL: https://github.com/apache/airflow/pull/7459
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039638#comment-17039638
 ] 

ASF subversion and git services commented on AIRFLOW-6820:
--

Commit ac45146dd21e64f5677c757dd4cf70ca834bbcc4 in airflow's branch 
refs/heads/v1-10-test from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ac45146 ]

[AIRFLOW-6820] split breeze into functions (#7433)

(cherry picked from commit 24f9f11775bf7b38cced9955f6c70705ad56f61c)


> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions

2020-02-18 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039639#comment-17039639
 ] 

ASF subversion and git services commented on AIRFLOW-6820:
--

Commit 453e65abcf9250685bf702b4018554e189d05ca9 in airflow's branch 
refs/heads/v1-10-test from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=453e65a ]

[AIRFLOW-6820] fixup! fix unbound variable

(cherry picked from commit a54cf8b33030a52c45e46cf7f507f496ee00d610)


> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6818) Prevent Docker cache-busting on when editing www templates

2020-02-18 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039635#comment-17039635
 ] 

ASF subversion and git services commented on AIRFLOW-6818:
--

Commit ee0ea017367cedc6bf2b9f17991e50d74cb78f38 in airflow's branch 
refs/heads/v1-10-test from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ee0ea01 ]

[AIRFLOW-6818] Prevent Docker cache-busting on when editing www templates 
(#7432)

There is two parts to this PR:

1. Only copying www/webpack.config.js and www/static/ before running the
   asset pipeline
2. Making sure that _all_ files (not just the critical ones) have the
   same permissions.

(cherry picked from commit cec9249f904bd91be9cb5a6be7e04979585da12e)


> Prevent Docker cache-busting on when editing www templates
> --
>
> Key: AIRFLOW-6818
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6818
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.10
>
>
> Due to permission differences on files/folders between Travis and docker hub 
> auto build we are needlessly busting the docker layer cache -> builds are a 
> bit slower than they need to be



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable

2020-02-18 Thread ASF subversion and git services (Jira)

potiuk commented on issue #7459: [AIRFLOW-6820] fixup! fix unbound variable
URL: https://github.com/apache/airflow/pull/7459#issuecomment-587996718
 
 
   Quick fix for unbound variable 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6816) Separate breeze's variable setup into functions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039636#comment-17039636
 ] 

ASF subversion and git services commented on AIRFLOW-6816:
--

Commit f6eafcefe9c61ef1dfdf29d0de1050feb3ed8fe4 in airflow's branch 
refs/heads/v1-10-test from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f6eafce ]

[AIRFLOW-6816] Simplified common functions in breeze scripts (#7430)

(cherry picked from commit ff7034f39ae4919ee8efbc8ef1c0feada1107d6a)


> Separate breeze's variable setup into functions
> ---
>
> Key: AIRFLOW-6816
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6816
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk opened a new pull request #7459: [AIRFLOW-6820] fixup! fix unbound variable

potiuk opened a new pull request #7459: [AIRFLOW-6820] fixup! fix unbound 
variable
URL: https://github.com/apache/airflow/pull/7459
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

mik-laj commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real 
subcommands for Breeze
URL: https://github.com/apache/airflow/pull/7458#issuecomment-587988274
 
 
   > also parsing options won't work either.
   
   Can you say something more about it?  I checked, I did found any issues
   https://user-images.githubusercontent.com/12058428/74792980-3e8ed280-52bf-11ea-8b55-1745beae606d.png;>
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6820) Split breeze into functions

2020-02-18 Thread ASF subversion and git services (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6820.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039601#comment-17039601
 ] 

ASF subversion and git services commented on AIRFLOW-6820:
--

Commit 24f9f11775bf7b38cced9955f6c70705ad56f61c in airflow's branch 
refs/heads/master from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=24f9f11 ]

[AIRFLOW-6820] split breeze into functions (#7433)



> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6820) Split breeze into functions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039600#comment-17039600
 ] 

ASF GitHub Bot commented on AIRFLOW-6820:
-

potiuk commented on pull request #7433: [AIRFLOW-6820] split breeze into 
functions
URL: https://github.com/apache/airflow/pull/7433
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Split breeze into functions
> ---
>
> Key: AIRFLOW-6820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6820
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: breeze
>Affects Versions: 2.0.0, 1.10.9
>Reporter: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7433: [AIRFLOW-6820] split breeze into functions

potiuk merged pull request #7433: [AIRFLOW-6820] split breeze into functions
URL: https://github.com/apache/airflow/pull/7433
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7433: [AIRFLOW-6820] split breeze into functions

codecov-io edited a comment on issue #7433: [AIRFLOW-6820] split breeze into 
functions
URL: https://github.com/apache/airflow/pull/7433#issuecomment-586737012
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=h1) 
Report
   > Merging 
[#7433](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9cbd7de6d115795aba8bfb8addb060bfdfbdf87b?src=pr=desc)
 will **increase** coverage by `0.25%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7433/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7433  +/-   ##
   ==
   + Coverage   86.37%   86.62%   +0.25% 
   ==
 Files 877  878   +1 
 Lines   4122141292  +71 
   ==
   + Hits3560335770 +167 
   + Misses   5618 5522  -96
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `87.78% <0%> (-0.23%)` | :arrow_down: |
   | 
[...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/config\_templates/default\_celery.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfY2VsZXJ5LnB5)
 | `52.94% <0%> (ø)` | :arrow_up: |
   | 
[...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `95.07% <0%> (ø)` | :arrow_up: |
   | 
[...low/providers/elasticsearch/hooks/elasticsearch.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZWxhc3RpY3NlYXJjaC9ob29rcy9lbGFzdGljc2VhcmNoLnB5)
 | `46.15% <0%> (ø)` | |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `98.29% <0%> (+0.01%)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `74.5% <0%> (+23.52%)` | :arrow_up: |
   | 
[...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==)
 | `94.89% <0%> (+25.51%)` | :arrow_up: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `92.25% <0%> (+45.07%)` | :arrow_up: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=footer). 
Last update 
[9cbd7de...c892ed2](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7433: [AIRFLOW-6820] split breeze into functions

potiuk commented on a change in pull request #7433: [AIRFLOW-6820] split breeze 
into functions
URL: https://github.com/apache/airflow/pull/7433#discussion_r381020777
 
 

 ##
 File path: hooks/build
 ##
 @@ -401,34 +372,30 @@ git ls-files -z -- ./ ':!:airflow/www/static/docs' | 
filterout_deleted_files | x
 # and this deals with directories
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

potiuk commented on a change in pull request #7458: [AIRFLOW-6838][WIP] 
Introduce real subcommands for Breeze
URL: https://github.com/apache/airflow/pull/7458#discussion_r381020442
 
 

 ##
 File path: breeze
 ##
 @@ -290,7 +290,7 @@ Usage: ${CMDNAME} [COMMAND] [FLAGS]  -- 
 
 The swiss-knife-army tool for Airflow testings. It allows to perform various 
test tasks:
 
-  * Enter interactive environment when no command flags are specified (default 
behaviour)
+  * Run interactive terminal when no command flags are specified (default 
behaviour)
 
 Review comment:
   IT's not terminal. It's the whole environment using integrations, runtime, 
kubernetes, databases etc. I still think environment is better name,


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands 
for Breeze
URL: https://github.com/apache/airflow/pull/7458#issuecomment-587978552
 
 
   While I like the spirit of that change,. Breeze grew a lot in terms of 
functionality and maybe we should think of refactoring the structure of 
commands soon.
   
   But I think it's not the right way of doing it.
   
   First of all you are just about to have a lot of conflicts as we are 
splitting breeze into functions. Secondly it does not take into account 
getopt's behaviour so it is for sure wrong (We define all the available options 
in breeze-complete and getopt by default assumes -- or - params. So not only 
autocomplete will not work but also parsing options won't work either.
   
   I think we need to do much bolder change if we want to introduce 
sub-commands - we should rewrite it in another language (python or golang). But 
complicating parsing of parameters is not a good idea. 
   
   Unless you use ready to use library - for example here 
http://people.tuebingen.mpg.de/maan/gsu/ then we could write it in  bash (but 
it's not a good idea as it is GPL2 and it does not support ZSH I believe). I 
think we should talk about it and think if that's the right time to think of 
such change or maybe better to leave it for after we deliver the current 
important 2.0 features - prod image, API and others. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7433: [AIRFLOW-6820] split breeze into functions

potiuk commented on issue #7433: [AIRFLOW-6820] split breeze into functions
URL: https://github.com/apache/airflow/pull/7433#issuecomment-587970268
 
 
   > Looks good other than one small tidy up.
   
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6838) Introduce real subcommands for Breeze



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039549#comment-17039549
 ] 

ASF GitHub Bot commented on AIRFLOW-6838:
-

mik-laj commented on pull request #7458: [AIRFLOW-6838] Introduce real 
subcommands for Breeze
URL: https://github.com/apache/airflow/pull/7458
 
 
   Breeze currently selects action using many different flags. I know from my 
own experience that this is very confusing. I think that subcommands will be 
much easier to use.
   
   I suspect that Breeze autocomplete is not working properly, but I have no 
experience with this component. Could someone help me with this?
   
   In the next step, I think that we should replace global variables with the 
function, because a large number of global variables makes it very difficult to 
trace the aapplication behavior.
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Introduce real subcommands for Breeze
> -
>
> Key: AIRFLOW-6838
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6838
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.9
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #7458: [AIRFLOW-6838] Introduce real subcommands for Breeze

mik-laj opened a new pull request #7458: [AIRFLOW-6838] Introduce real 
subcommands for Breeze
URL: https://github.com/apache/airflow/pull/7458
 
 
   Breeze currently selects action using many different flags. I know from my 
own experience that this is very confusing. I think that subcommands will be 
much easier to use.
   
   I suspect that Breeze autocomplete is not working properly, but I have no 
experience with this component. Could someone help me with this?
   
   In the next step, I think that we should replace global variables with the 
function, because a large number of global variables makes it very difficult to 
trace the aapplication behavior.
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7433: [AIRFLOW-6820] split breeze into functions

ashb commented on a change in pull request #7433: [AIRFLOW-6820] split breeze 
into functions
URL: https://github.com/apache/airflow/pull/7433#discussion_r380991455
 
 

 ##
 File path: hooks/build
 ##
 @@ -401,34 +372,30 @@ git ls-files -z -- ./ ':!:airflow/www/static/docs' | 
filterout_deleted_files | x
 # and this deals with directories
 
 Review comment:
   Can you remove the exclusion I added on the previous `git ls-files` lines 
now you've removed the symlink please?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-6837) limit description of dag in home page

2020-02-18 Thread Ping Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ping Zhang reassigned AIRFLOW-6837:
---

Assignee: Ping Zhang

> limit description of dag in home page
> -
>
> Key: AIRFLOW-6837
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6837
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.9
>Reporter: Ping Zhang
>Assignee: Ping Zhang
>Priority: Major
>
> so that the tooltip will not cover the dag link



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6838) Introduce real subcommands for Breeze

2020-02-18 Thread Kamil Bregula (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula updated AIRFLOW-6838:
---
Summary: Introduce real subcommands for Breeze  (was: Introdocue real 
subcommands for Breeze)

> Introduce real subcommands for Breeze
> -
>
> Key: AIRFLOW-6838
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6838
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.9
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6837) limit description of dag in home page



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039542#comment-17039542
 ] 

ASF GitHub Bot commented on AIRFLOW-6837:
-

pingzh commented on pull request #7457: [AIRFLOW-6837] limit description len of 
dag in home
URL: https://github.com/apache/airflow/pull/7457
 
 
   as it makes dags link not clickable
   
   
   The dag with long description:
   
![image](https://user-images.githubusercontent.com/8662365/74785686-bb8a5f80-525f-11ea-8463-af10b3c1ad40.png)
   
   Before:
   
![image](https://user-images.githubusercontent.com/8662365/74785702-c6dd8b00-525f-11ea-94af-9710da21baab.png)
   
   After:
   
   
![image](https://user-images.githubusercontent.com/8662365/74785710-ca711200-525f-11ea-9e12-6a77ebc9bcad.png)
   
   
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> limit description of dag in home page
> -
>
> Key: AIRFLOW-6837
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6837
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.9
>Reporter: Ping Zhang
>Priority: Major
>
> so that the tooltip will not cover the dag link



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6838) Introdocue real subcommands for Breeze

2020-02-18 Thread Kamil Bregula (Jira)

Kamil Bregula created AIRFLOW-6838:
--

 Summary: Introdocue real subcommands for Breeze
 Key: AIRFLOW-6838
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6838
 Project: Apache Airflow
  Issue Type: Bug
  Components: core
Affects Versions: 1.10.9
Reporter: Kamil Bregula






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] pingzh opened a new pull request #7457: [AIRFLOW-6837] limit description len of dag in home

pingzh opened a new pull request #7457: [AIRFLOW-6837] limit description len of 
dag in home
URL: https://github.com/apache/airflow/pull/7457
 
 
   as it makes dags link not clickable
   
   
   The dag with long description:
   
![image](https://user-images.githubusercontent.com/8662365/74785686-bb8a5f80-525f-11ea-8463-af10b3c1ad40.png)
   
   Before:
   
![image](https://user-images.githubusercontent.com/8662365/74785702-c6dd8b00-525f-11ea-94af-9710da21baab.png)
   
   After:
   
   
![image](https://user-images.githubusercontent.com/8662365/74785710-ca711200-525f-11ea-9e12-6a77ebc9bcad.png)
   
   
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6837) limit description of dag in home page

2020-02-18 Thread Ping Zhang (Jira)

Ping Zhang created AIRFLOW-6837:
---

 Summary: limit description of dag in home page
 Key: AIRFLOW-6837
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6837
 Project: Apache Airflow
  Issue Type: Improvement
  Components: webserver
Affects Versions: 1.10.9
Reporter: Ping Zhang


so that the tooltip will not cover the dag link



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] msb217 opened a new pull request #7456: [AIRFLOW 6817] - Move airflow init.py imports to sub-packages

msb217 opened a new pull request #7456: [AIRFLOW 6817]  - Move airflow 
__init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Moved `airflow/__init__.py` imports to respective sub-packages with the 
exception of the `settings.initialize()` and `integrate_plugins()`
   
   All breeze unit and integration tests are passing locally.
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator

codecov-io edited a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy REST 
operator
URL: https://github.com/apache/airflow/pull/6090#issuecomment-572009682
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=h1) 
Report
   > Merging 
[#6090](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/7c95c8144d1262a5d7e9d99ca1cec30941064c24?src=pr=desc)
 will **decrease** coverage by `0.41%`.
   > The diff coverage is `98.6%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6090/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6090  +/-   ##
   ==
   - Coverage   86.66%   86.25%   -0.42% 
   ==
 Files 877  882   +5 
 Lines   4125241507 +255 
   ==
   + Hits3575135800  +49 
   - Misses   5501 5707 +206
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `95.07% <ø> (ø)` | :arrow_up: |
   | 
[airflow/providers/apache/livy/sensors/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvc2Vuc29ycy9saXZ5LnB5)
 | `100% <100%> (ø)` | |
   | 
[...providers/apache/livy/example\_dags/example\_livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvZXhhbXBsZV9kYWdzL2V4YW1wbGVfbGl2eS5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `98.3% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/providers/apache/livy/operators/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvb3BlcmF0b3JzL2xpdnkucHk=)
 | `97.5% <97.5%> (ø)` | |
   | 
[airflow/providers/apache/livy/hooks/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvaG9va3MvbGl2eS5weQ==)
 | `98.62% <98.62%> (ø)` | |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `56% <0%> (-44%)` | :arrow_down: |
   | ... and [12 
more](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=footer). 
Last update 
[7c95c81...2b0e5e2](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow

vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add 
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587936273
 
 
   Actually it looks like it wasn't happy with working_dir as a kwargs, I'll 
try adding it to the type definitions to see if that changes anything...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator

codecov-io edited a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy REST 
operator
URL: https://github.com/apache/airflow/pull/6090#issuecomment-572009682
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=h1) 
Report
   > Merging 
[#6090](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/7c95c8144d1262a5d7e9d99ca1cec30941064c24?src=pr=desc)
 will **decrease** coverage by `0.6%`.
   > The diff coverage is `98.6%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6090/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6090  +/-   ##
   ==
   - Coverage   86.66%   86.05%   -0.61% 
   ==
 Files 877  882   +5 
 Lines   4125241507 +255 
   ==
   - Hits3575135720  -31 
   - Misses   5501 5787 +286
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `95.07% <ø> (ø)` | :arrow_up: |
   | 
[airflow/providers/apache/livy/sensors/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvc2Vuc29ycy9saXZ5LnB5)
 | `100% <100%> (ø)` | |
   | 
[...providers/apache/livy/example\_dags/example\_livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvZXhhbXBsZV9kYWdzL2V4YW1wbGVfbGl2eS5weQ==)
 | `100% <100%> (ø)` | |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `98.3% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/providers/apache/livy/operators/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvb3BlcmF0b3JzL2xpdnkucHk=)
 | `97.5% <97.5%> (ø)` | |
   | 
[airflow/providers/apache/livy/hooks/livy.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvaG9va3MvbGl2eS5weQ==)
 | `98.62% <98.62%> (ø)` | |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0%> (-64.16%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | ... and [18 
more](https://codecov.io/gh/apache/airflow/pull/6090/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=footer). 
Last update 
[7c95c81...2b0e5e2](https://codecov.io/gh/apache/airflow/pull/6090?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow

vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add 
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587935001
 
 
   Can somebody explain what the task_id (provided as task-id?) is expected to 
work, be parsed, etc? @mik-laj added it to the test cases last year, and it's 
not clear if I need to be parsing this for the operator - just trying as it was 
it failed that the key word argument wasn't valid.
   
   >airflow.exceptions.AirflowException: Invalid arguments were 
passed to SingularityOperator (task_id: task-id). Invalid arguments were:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6836) DebugExecutor failing to change task state.

2020-02-18 Thread Sam Wheating (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Wheating updated AIRFLOW-6836:
--
Description: 
Running a DAG locally with the DebugExecutor fails with the following error:

 

{noformat}
  File 
"/Users/samwheating/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/executors/debug_executor.py",
 line 148, in change_state
 self.running.remove(key)
 AttributeError: 'dict' object has no attribute 'remove'
{noformat}

This seems to be because the change_state function is expecting a set rather 
than a dict for `self.running`. This should be updated to use `del 
self.running[key]` or `self.running.pop(key)`.

I'll submit a PR for this tomorrow. 

  was:
Running a DAG locally with the DebugExecutor fails with the following error:

```
 File 
"/Users/samwheating/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/executors/debug_executor.py",
 line 148, in change_state
 self.running.remove(key)
AttributeError: 'dict' object has no attribute 'remove'
```

This seems to be because the change_state function is expecting a set rather 
than a dict for `self.running`. This should be updated to use `del 
self.running[key]` or `self.running.pop(key)`.

I'll submit a PR for this tomorrow. 


> DebugExecutor failing to change task state.
> ---
>
> Key: AIRFLOW-6836
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6836
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executors
>Affects Versions: 1.10.9
>Reporter: Sam Wheating
>Assignee: Sam Wheating
>Priority: Minor
>  Labels: easyfix
> Fix For: 1.10.10
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Running a DAG locally with the DebugExecutor fails with the following error:
>  
> {noformat}
>   File 
> "/Users/samwheating/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/executors/debug_executor.py",
>  line 148, in change_state
>  self.running.remove(key)
>  AttributeError: 'dict' object has no attribute 'remove'
> {noformat}
> This seems to be because the change_state function is expecting a set rather 
> than a dict for `self.running`. This should be updated to use `del 
> self.running[key]` or `self.running.pop(key)`.
> I'll submit a PR for this tomorrow. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6836) DebugExecutor failing to change task state.

2020-02-18 Thread Sam Wheating (Jira)

Sam Wheating created AIRFLOW-6836:
-

 Summary: DebugExecutor failing to change task state.
 Key: AIRFLOW-6836
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6836
 Project: Apache Airflow
  Issue Type: Bug
  Components: executors
Affects Versions: 1.10.9
Reporter: Sam Wheating
Assignee: Sam Wheating
 Fix For: 1.10.10


Running a DAG locally with the DebugExecutor fails with the following error:

```
 File 
"/Users/samwheating/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/executors/debug_executor.py",
 line 148, in change_state
 self.running.remove(key)
AttributeError: 'dict' object has no attribute 'remove'
```

This seems to be because the change_state function is expecting a set rather 
than a dict for `self.running`. This should be updated to use `del 
self.running[key]` or `self.running.pop(key)`.

I'll submit a PR for this tomorrow. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs cli command should allow exec_date between start/end range and print start/end times

ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs 
cli command should allow exec_date between start/end range and print start/end 
times
URL: https://github.com/apache/airflow/pull/7446#discussion_r380961616
 
 

 ##
 File path: airflow/models/dagrun.py
 ##
 @@ -135,7 +135,7 @@ def find(dag_id=None, run_id=None, execution_date=None,
 :param run_id: defines the run id for this dag run
 :type run_id: str
 :param execution_date: the execution date
-:type execution_date: datetime.datetime
+:type execution_date: datetime.datetime, list
 
 Review comment:
   ```suggestion
   :type execution_date: datetime.datetime or list[datetime.datetime]
   ```
   
   Then


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs cli command should allow exec_date between start/end range and print start/end times

ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs 
cli command should allow exec_date between start/end range and print start/end 
times
URL: https://github.com/apache/airflow/pull/7446#discussion_r380961251
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -597,8 +608,12 @@ class CLIFactory:
 'help': "List dag runs given a DAG id. If state option is given, 
it will only "
 "search for all the dagruns with the given state. "
 "If no_backfill option is given, it will filter out "
-"all backfill dagruns for given dag id",
-'args': ('dag_id', 'no_backfill', 'state', 'output',),
+"all backfill dagruns for given dag id"
+"If exec_date_from is given, it will filter out "
+"all the dagruns that were executed before this date. "
+"If exec_date_to is given, it will filter out "
+"all the dagruns that were executed after this date. ",
+'args': ('dag_id_opt', 'no_backfill', 'state', 'output', 
"exec_date_from", "exec_date_to"),
 
 Review comment:
   `airflow backfill` has `--start_date` and `--end_date` args that serve a 
similar purpose. I think you should re-used those names instead of creating new 
ones.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs cli command should allow exec_date between start/end range and print start/end times

ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs 
cli command should allow exec_date between start/end range and print start/end 
times
URL: https://github.com/apache/airflow/pull/7446#discussion_r380960677
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -597,8 +608,12 @@ class CLIFactory:
 'help': "List dag runs given a DAG id. If state option is given, 
it will only "
 "search for all the dagruns with the given state. "
 "If no_backfill option is given, it will filter out "
-"all backfill dagruns for given dag id",
-'args': ('dag_id', 'no_backfill', 'state', 'output',),
+"all backfill dagruns for given dag id"
+"If exec_date_from is given, it will filter out "
+"all the dagruns that were executed before this date. "
+"If exec_date_to is given, it will filter out "
+"all the dagruns that were executed after this date. ",
 
 Review comment:
   What does this look like when you run `airflow dag_runs --list`? I think you 
have formatting "bugs" in here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs cli command should allow exec_date between start/end range and print start/end times

ashb commented on a change in pull request #7446: [AIRFLOW-6474] list_dag_runs 
cli command should allow exec_date between start/end range and print start/end 
times
URL: https://github.com/apache/airflow/pull/7446#discussion_r380960239
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -131,11 +131,22 @@ class CLIFactory:
 # list_dag_runs
 'no_backfill': Arg(
 ("--no_backfill",),
-"filter all the backfill dagruns given the dag id", "store_true"),
 
 Review comment:
   It's good practice to avoid whitespace/formatting only changes to code you 
don't touch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6474) list_dag_runs cli command should allow exec_date between start/end range and print start/end times

2020-02-18 Thread alexander (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039471#comment-17039471
 ] 

alexander commented on AIRFLOW-6474:


I've added printing of end_date.

> list_dag_runs cli command should allow exec_date between start/end range and 
> print start/end times
> --
>
> Key: AIRFLOW-6474
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6474
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: alexander
>Priority: Major
>  Labels: gsoc, gsoc2020, mentor
>
> 1. accept argument exec_date_from, exec_date_to to filter execution_dates 
> returned, ie show dag runs with exec_date between 20190901 and 20190930
> 2. separate to that in the output print the start_date and end_date of each 
> dagrun (ie execdate for 20190907 had start_date 2019090804:23 and end_date  
> 2019090804:38
> 3. dag_id arg should be optional



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare 
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380941844
 
 

 ##
 File path: README.md
 ##
 @@ -107,6 +108,33 @@ unit of work and continuity.
   ![](/docs/img/code.png)
 
 
+## Airflow Versions, backporting packages and future upgrade path
+
+
+Currently stable versions of Apache Airflow are released in 1.10.* series. We 
are working on the
+future, incompatible version of Airflow from the 2.0.* series. It is going to 
be released in
+in 2020. However the exact time of release depends on many factors and is yet 
unknown.
+We have already a lot of changes in the hooks/operators/sensors for many 
external systems
+and they are not used because they are part of the master/2.0 release.
+
+In the Airflow 2.0 - following AIP-21 "change in import paths" all the 
non-core operators/hooks/sensors
+of Apache Airflow have been moved to the "airflow.providers" package. This 
opened a possibility to
+use the operators from Airflow 2.0 in Airflow 1.10 - with the constraint that 
those
+packages can only be used in python3.6+ environment.
+
+Therefore we decided to prepare and release backport packages that can be 
installed
+for older Airflow versions. Those backport packages are released more 
frequently. Users do not
+have to upgrade their Airflow version to use those packages. There are a 
number of incompatibilities
+between Airflow 2.0 and 1.10.* - documented in [UPDATING.md](UPDATING.md). 
With backported
+providers package users can migrate their DAGs to the new providers package 
incrementally
+and once they convert to the new operators/sensors/hooks they can seamlessly 
migrate their
+environments to Airflow 2.0.
+
+More information about the status and releases of the back-ported packages are 
available
+at 
https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
+
+You
 
 Review comment:
   ```suggestion
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare 
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380941290
 
 

 ##
 File path: README.md
 ##
 @@ -107,6 +108,33 @@ unit of work and continuity.
   ![](/docs/img/code.png)
 
 
+## Airflow Versions, backporting packages and future upgrade path
 
 Review comment:
   ```suggestion
   ## Using hooks and Operators from "master" in Airflow 1.10
   ```
   
   I think this captures what users' would want to see more clearly in the ToC


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare 
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380940650
 
 

 ##
 File path: README.md
 ##
 @@ -107,6 +108,33 @@ unit of work and continuity.
   ![](/docs/img/code.png)
 
 
+## Airflow Versions, backporting packages and future upgrade path
+
+
+Currently stable versions of Apache Airflow are released in 1.10.* series. We 
are working on the
+future, incompatible version of Airflow from the 2.0.* series. It is going to 
be released in
 
 Review comment:
   Let's not say "incompatible" like this please -- that's a stronger word than 
we need.
   
   ```
   Currently stable versions of Apache Airflow are released in 1.10.* series. 
We are working on the
   future, Airflow 2.x series which has some backwards-incompatible changes
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare 
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380939740
 
 

 ##
 File path: MANIFEST-packages.in
 ##
 @@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
 
 Review comment:
   This is warning is still showing up on CI @potiuk 
   
   
   ```
   
---
   
Preparing backporting package mongo
   
   
---
   
   warning: Failed to find the configured license file 'licenses/*'
   
   warning: no previously-included files matching '__pycache__' found anywhere 
in distribution
   
   warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
   
   warning: check: missing required meta-data: url
   
   warning: build_py: byte-compiling is disabled, skipping.
   
   warning: install_lib: byte-compiling is disabled, skipping.
   
   gitpython not found: Cannot compute the git version.
   
   running clean
   
   removing 'build/lib' (and everything under it)
   
   removing 'build/bdist.linux-x86_64' (and everything under it)
   
   'build/scripts-3.6' does not exist -- can't clean it
   
   removing 'build'
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare 
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380939740
 
 

 ##
 File path: MANIFEST-packages.in
 ##
 @@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
 
 Review comment:
   This is warning is still showing up on CI @potiuk 
   
   https://travis-ci.org/apache/airflow/jobs/651778738
   
   ```
   
---
   
Preparing backporting package mongo
   
   
---
   
   warning: Failed to find the configured license file 'licenses/*'
   
   warning: no previously-included files matching '__pycache__' found anywhere 
in distribution
   
   warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
   
   warning: check: missing required meta-data: url
   
   warning: build_py: byte-compiling is disabled, skipping.
   
   warning: install_lib: byte-compiling is disabled, skipping.
   
   gitpython not found: Cannot compute the git version.
   
   running clean
   
   removing 'build/lib' (and everything under it)
   
   removing 'build/bdist.linux-x86_64' (and everything under it)
   
   'build/scripts-3.6' does not exist -- can't clean it
   
   removing 'build'
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages

ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-587860742
 
 
   :eyes: 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1202) Create Elasticsearch Hook

2020-02-18 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039456#comment-17039456
 ] 

ASF GitHub Bot commented on AIRFLOW-1202:
-

potiuk commented on pull request #7358: [AIRFLOW-1202] Create Elasticsearch Hook
URL: https://github.com/apache/airflow/pull/7358
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create Elasticsearch Hook 
> --
>
> Key: AIRFLOW-1202
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1202
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Hesen Peng
>Assignee: Hesen Peng
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> i would like to add an Elasticsearch hook for Airflow. The code is already 
> done in the branch linked. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-1202) Create Elasticsearch Hook



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039457#comment-17039457
 ] 

ASF subversion and git services commented on AIRFLOW-1202:
--

Commit 65dd28eb77d996ec8306c67d5ce1ccee2c14cc9d in airflow's branch 
refs/heads/master from Hossein Torabi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=65dd28e ]

[AIRFLOW-1202] Create Elasticsearch Hook (#7358)



> Create Elasticsearch Hook 
> --
>
> Key: AIRFLOW-1202
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1202
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Hesen Peng
>Assignee: Hesen Peng
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> i would like to add an Elasticsearch hook for Airflow. The code is already 
> done in the branch linked. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-1202) Create Elasticsearch Hook



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-1202.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Create Elasticsearch Hook 
> --
>
> Key: AIRFLOW-1202
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1202
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Hesen Peng
>Assignee: Hesen Peng
>Priority: Minor
> Fix For: 2.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> i would like to add an Elasticsearch hook for Airflow. The code is already 
> done in the branch linked. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook

potiuk commented on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook
URL: https://github.com/apache/airflow/pull/7358#issuecomment-587844278
 
 
   Thanks @blcksrx !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #7358: [AIRFLOW-1202] Create Elasticsearch Hook

potiuk merged pull request #7358: [AIRFLOW-1202] Create Elasticsearch Hook
URL: https://github.com/apache/airflow/pull/7358
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6835) GoogleCloudStorageToBigQueryOperator failed when loading empty csv file

2020-02-18 Thread Feng Gao (Jira)

Feng Gao created AIRFLOW-6835:
-

 Summary: GoogleCloudStorageToBigQueryOperator failed when loading 
empty csv file
 Key: AIRFLOW-6835
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6835
 Project: Apache Airflow
  Issue Type: Bug
  Components: gcp
Affects Versions: 1.10.6
 Environment: Google Composer Image version: 
composer-1.9.0-airflow-1.10.6
Reporter: Feng Gao


GoogleCloudStorageToBigQueryOperator failed when loading empty csv file (size 
0).

Found error in logs : 

Exception: BigQuery job failed. Final error was: \{'reason': 'invalidQuery', 
'location': 'query', 'message': 'There are no column descriptions provided for 
table _ev_5GNTR3nPyPPZfamjrtfpgieU8fg6kU3A_8e8qCoEAMI_source'}.

Re-ran the task succeeded after adding one row to the csv file.

 

Google Composer Image version: composer-1.9.0-airflow-1.10.6

 

The same dag works fine on Google Composer image version: 
composer-1.8.4-airflow-1.9.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow

vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add 
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587823490
 
 
   docs build passed! Tiny progress :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages

potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-587815757
 
 
   @ashb @mik-laj ?  Can I merge the backporting packages?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7433: [AIRFLOW-6820] split breeze into functions

potiuk commented on issue #7433: [AIRFLOW-6820] split breeze into functions
URL: https://github.com/apache/airflow/pull/7433#issuecomment-587814433
 
 
   Hey @ashb. Are you ok with how it is ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7433: [AIRFLOW-6820] split breeze into functions

codecov-io edited a comment on issue #7433: [AIRFLOW-6820] split breeze into 
functions
URL: https://github.com/apache/airflow/pull/7433#issuecomment-586737012
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=h1) 
Report
   > Merging 
[#7433](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9cbd7de6d115795aba8bfb8addb060bfdfbdf87b?src=pr=desc)
 will **increase** coverage by `0.23%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7433/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #7433  +/-   ##
   =
   + Coverage   86.37%   86.6%   +0.23% 
   =
 Files 877 877  
 Lines   41221   41336 +115 
   =
   + Hits35603   35799 +196 
   + Misses   56185537  -81
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `82.03% <0%> (-5.98%)` | :arrow_down: |
   | 
[...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/config\_templates/default\_celery.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfY2VsZXJ5LnB5)
 | `54.16% <0%> (+1.22%)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `74.5% <0%> (+23.52%)` | :arrow_up: |
   | 
[...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==)
 | `94.89% <0%> (+25.51%)` | :arrow_up: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `92.25% <0%> (+45.07%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `100% <0%> (+47.05%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7433/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `100% <0%> (+55.55%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=footer). 
Last update 
[9cbd7de...84fce4b](https://codecov.io/gh/apache/airflow/pull/7433?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7455: [AIRFLOW-XXXX] Change title and add some more helpful information

potiuk commented on issue #7455: [AIRFLOW-] Change title and add some more 
helpful information
URL: https://github.com/apache/airflow/pull/7455#issuecomment-587766510
 
 
   Thanks @ryw 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #7455: [AIRFLOW-XXXX] Change title and add some more helpful information

potiuk merged pull request #7455: [AIRFLOW-] Change title and add some more 
helpful information
URL: https://github.com/apache/airflow/pull/7455
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for 
backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-587763532
 
 
   Applied changes from @ashb comments in #7439 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] snazzyfox commented on a change in pull request #7451: [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator

snazzyfox commented on a change in pull request #7451: [AIRFLOW-6830] Add 
Subject/MessageAttributes to SNS hook and operator
URL: https://github.com/apache/airflow/pull/7451#discussion_r380856952
 
 

 ##
 File path: tests/providers/amazon/aws/hooks/test_sns.py
 ##
 @@ -49,7 +49,24 @@ def test_publish_to_target(self):
 self.assertTrue('MessageId' in response)
 
 @mock_sns
-def test_publish_to_target_without_subject(self):
+def test_publish_to_target_with_attributes(self):
+hook = AwsSnsHook(aws_conn_id='aws_default')
+
+message = "Hello world"
+topic_name = "test-topic"
+target = hook.get_conn().create_topic(Name=topic_name).get('TopicArn')
+
+response = hook.publish_to_target(target, message, message_attributes={
+'test-string': 'string-value',
+'test-number': 123456,
+'test-array': ['first', 'second', 'third'],
+'test-binary': b'binary-value',
+})
+
+self.assertTrue('MessageId' in response)
 
 Review comment:
   I used assertTrue since that's used in the rest of these tests. ~~Should I 
update those as well?~~ I've updated the untouched tests in this file as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6255) Redis Hook - refactor & add most important methods



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039412#comment-17039412
 ] 

ASF GitHub Bot commented on AIRFLOW-6255:
-

stale[bot] commented on pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Redis Hook - refactor & add most important methods
> --
>
> Key: AIRFLOW-6255
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6255
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.6
>Reporter: Mariusz Górski
>Assignee: Mariusz Górski
>Priority: Minor
>  Labels: hooks, redis, refactor
>
> Currently Redis Hook consists of get_conn func only and every operation using 
> it is calling Redis client methods directly. This Issue aims to extend Redis 
> Hook so it provides all necessary methods wrapping Redis functionality used 
> so far.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #6820: [AIRFLOW-6255] Redis Hook Refactor

stale[bot] closed pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7446: [AIRFLOW-6474] list_dag_runs cli command should allow exec_date between start/end range and print start/end times

codecov-io edited a comment on issue #7446: [AIRFLOW-6474] list_dag_runs cli 
command should allow exec_date between start/end range and print start/end times
URL: https://github.com/apache/airflow/pull/7446#issuecomment-587128695
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=h1) 
Report
   > Merging 
[#7446](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d1ab3705d3d03f68abc2b900ebb35e8497dee2e7?src=pr=desc)
 will **decrease** coverage by `0.44%`.
   > The diff coverage is `71.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7446/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7446  +/-   ##
   ==
   - Coverage   86.64%   86.19%   -0.45% 
   ==
 Files 878  877   -1 
 Lines   4121941258  +39 
   ==
   - Hits3571535564 -151 
   - Misses   5504 5694 +190
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `94.73% <ø> (ø)` | :arrow_up: |
   | 
[airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=)
 | `84.41% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5)
 | `95.79% <66.66%> (-0.76%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0%> (-64.16%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0%> (-45.66%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55% <0%> (-45%)` | :arrow_down: |
   | 
[...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==)
 | `69.38% <0%> (-25.52%)` | :arrow_down: |
   | ... and [33 
more](https://codecov.io/gh/apache/airflow/pull/7446/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=footer). 
Last update 
[d1ab370...886fca4](https://codecov.io/gh/apache/airflow/pull/7446?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7439: [AIRFLOW-6204][depends on 6763] Create GoogleSystemTest class

2020-02-18 Thread ASF subversion and git services (Jira)

potiuk commented on a change in pull request #7439: [AIRFLOW-6204][depends on 
6763] Create GoogleSystemTest class
URL: https://github.com/apache/airflow/pull/7439#discussion_r380888600
 
 

 ##
 File path: tests/conftest.py
 ##
 @@ -87,6 +88,26 @@ def pytest_addoption(parser):
 metavar="RUNTIME",
 help="only run tests matching the runtime: [kubernetes].",
 )
+group.addoption(
+"--systems",
+action="store",
+metavar="SYSTEMS",
+help="only run tests matching the systems specified [google.cloud, 
google.marketing_platform]",
+)
+group.addoption(
+"--include-long-lasting",
 
 Review comment:
   Updated in my change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6823) Breeze broken on current master on Linux



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039384#comment-17039384
 ] 

ASF subversion and git services commented on AIRFLOW-6823:
--

Commit 03fb50aaa4bd7b00968c300781039cebf91043f2 in airflow's branch 
refs/heads/master from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=03fb50a ]

[AIRFLOW-6823] Fix root owned files automatically in Breeze (#7449)



> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 2.0.0
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6823) Breeze broken on current master on Linux



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039383#comment-17039383
 ] 

ASF GitHub Bot commented on AIRFLOW-6823:
-

potiuk commented on pull request #7449: [AIRFLOW-6823] Fix root owned files 
automatically in Breeze
URL: https://github.com/apache/airflow/pull/7449
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 2.0.0
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6823) Breeze broken on current master on Linux



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6823.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Breeze broken on current master on Linux
> 
>
> Key: AIRFLOW-6823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6823
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: breeze
>Affects Versions: 2.0.0
>Reporter: Aleksander Nitecki
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Breeze on current master log.txt
>
>
> See the attachment for log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7449: [AIRFLOW-6823] Fix root owned files automatically in Breeze

potiuk merged pull request #7449: [AIRFLOW-6823] Fix root owned files 
automatically in Breeze
URL: https://github.com/apache/airflow/pull/7449
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7451: [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator