[jira] [Resolved] (AIRFLOW-1557) backfill ignores configured number of slots in a pool
[
https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-1557.
---
Resolution: Fixed
Fix Version/s: 1.10.3
Thanks [~milton0825]. I think this pr should be included in 1.10.3.
> backfill ignores configured number of slots in a pool
> -
>
> Key: AIRFLOW-1557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1557
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ash Berlin-Taylor
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.3
>
> Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at
> 11.39.32.png
>
>
> I ran a backfill command with this tool:
> The backfill process appears to run as many tasks as possible, even when the
> pool it is running in should limit the numbers:
> {noformat}
> airflow backfill \
> -t fetch_dk_unfiltered \
> --pool brand_index_api \
> -s 2017-07-31 -e 2017-08-31 \
> -x \
> brand_index_fetcher
> {noformat}
> (Nothing else than the backfill is currently using this pool. I wasn't able
> to capture a screen shot of the task instances before the jobs completed.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
[ https://issues.apache.org/jira/browse/AIRFLOW-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2652. --- Resolution: Fixed Fix Version/s: 1.10.3 > implement / enhance pythonOperator deepcopy > --- > > Key: AIRFLOW-2652 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > > Currently if we backfill an airflow task which is pythonOperator, when > airflow does a deepcopy of callable from the operator we may encounter > unexpected issue. We should skip the callable deepcopy as it will not get > mutated. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-2558.
---
Resolution: Fixed
Fix Version/s: (was: 1.10.0)
1.10.3
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.3
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1557) backfill ignores configured number of slots in a pool
[
https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799904#comment-16799904
]
ASF subversion and git services commented on AIRFLOW-1557:
--
Commit c15dad3480f198cc3461f5e469f6a1ce88ea17ea in airflow's branch
refs/heads/master from Chao-Han Tsai
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c15dad3 ]
[AIRFLOW-1557] Backfill should respect pool (#4949)
The number of backfill tasks should be limited
by either non_pooled_backfill_task_slot_count
or the open slots of pool.
> backfill ignores configured number of slots in a pool
> -
>
> Key: AIRFLOW-1557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1557
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ash Berlin-Taylor
>Assignee: Chao-Han Tsai
>Priority: Major
> Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at
> 11.39.32.png
>
>
> I ran a backfill command with this tool:
> The backfill process appears to run as many tasks as possible, even when the
> pool it is running in should limit the numbers:
> {noformat}
> airflow backfill \
> -t fetch_dk_unfiltered \
> --pool brand_index_api \
> -s 2017-07-31 -e 2017-08-31 \
> -x \
> brand_index_fetcher
> {noformat}
> (Nothing else than the backfill is currently using this pool. I wasn't able
> to capture a screen shot of the task instances before the jobs completed.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2537) clearing tasks shouldn't set backfill DAG runs to `running`
[ https://issues.apache.org/jira/browse/AIRFLOW-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2537. --- Resolution: Fixed Fix Version/s: 1.10.3 > clearing tasks shouldn't set backfill DAG runs to `running` > --- > > Key: AIRFLOW-2537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2537 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reopened AIRFLOW-2558:
---
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.0
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2537) clearing tasks shouldn't set backfill DAG runs to `running`
[ https://issues.apache.org/jira/browse/AIRFLOW-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reopened AIRFLOW-2537: --- > clearing tasks shouldn't set backfill DAG runs to `running` > --- > > Key: AIRFLOW-2537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2537 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
[ https://issues.apache.org/jira/browse/AIRFLOW-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reopened AIRFLOW-2652: --- > implement / enhance pythonOperator deepcopy > --- > > Key: AIRFLOW-2652 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Currently if we backfill an airflow task which is pythonOperator, when > airflow does a deepcopy of callable from the operator we may encounter > unexpected issue. We should skip the callable deepcopy as it will not get > mutated. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1557) backfill ignores configured number of slots in a pool
[
https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799903#comment-16799903
]
ASF GitHub Bot commented on AIRFLOW-1557:
-
feng-tao commented on pull request #4949: [AIRFLOW-1557] Backfill should
respect pool limit
URL: https://github.com/apache/airflow/pull/4949
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> backfill ignores configured number of slots in a pool
> -
>
> Key: AIRFLOW-1557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1557
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ash Berlin-Taylor
>Assignee: Chao-Han Tsai
>Priority: Major
> Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at
> 11.39.32.png
>
>
> I ran a backfill command with this tool:
> The backfill process appears to run as many tasks as possible, even when the
> pool it is running in should limit the numbers:
> {noformat}
> airflow backfill \
> -t fetch_dk_unfiltered \
> --pool brand_index_api \
> -s 2017-07-31 -e 2017-08-31 \
> -x \
> brand_index_fetcher
> {noformat}
> (Nothing else than the backfill is currently using this pool. I wasn't able
> to capture a screen shot of the task instances before the jobs completed.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit
feng-tao merged pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit URL: https://github.com/apache/airflow/pull/4949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3982) DagRun state updater is incorrect when DagRun's tasks don't correspond to DAG's
[ https://issues.apache.org/jira/browse/AIRFLOW-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3982. --- Resolution: Fixed Fix Version/s: 1.10.3 > DagRun state updater is incorrect when DagRun's tasks don't correspond to > DAG's > --- > > Key: AIRFLOW-3982 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3982 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dima Kamalov >Assignee: Dima Kamalov >Priority: Minor > Fix For: 1.10.3 > > > To repro: > (1) create a Dag with a start date and a task, run it > (2) add a task in that DAG with a later start date > (3) rerun past run – tasks will get processed correctly but DagRun state > won't get updated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] mik-laj commented on issue #4823: [AIRFLOW-3999] Remove all inline style
mik-laj commented on issue #4823: [AIRFLOW-3999] Remove all inline style URL: https://github.com/apache/airflow/pull/4823#issuecomment-475926199 @ashb I updated PR. Should I change the class names for elements related to the legend on non-BEM versions? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3982) DagRun state updater is incorrect when DagRun's tasks don't correspond to DAG's
[ https://issues.apache.org/jira/browse/AIRFLOW-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799894#comment-16799894 ] ASF subversion and git services commented on AIRFLOW-3982: -- Commit bbcaf29e698d6d1eae132996fb32dbd77123c4f3 in airflow's branch refs/heads/master from Dima Kamalov [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bbcaf29 ] [AIRFLOW-3982] Update DagRun state based on its own tasks (#4808) > DagRun state updater is incorrect when DagRun's tasks don't correspond to > DAG's > --- > > Key: AIRFLOW-3982 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3982 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dima Kamalov >Assignee: Dima Kamalov >Priority: Minor > > To repro: > (1) create a Dag with a start date and a task, run it > (2) add a task in that DAG with a later start date > (3) rerun past run – tasks will get processed correctly but DagRun state > won't get updated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's
feng-tao merged pull request #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's URL: https://github.com/apache/airflow/pull/4808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao merged pull request #4964: [AIRFLOW-XXX] Add to the list of engineers working on Airflow at Bombora Inc
feng-tao merged pull request #4964: [AIRFLOW-XXX] Add to the list of engineers working on Airflow at Bombora Inc URL: https://github.com/apache/airflow/pull/4964 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels
mik-laj edited a comment on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels URL: https://github.com/apache/airflow/pull/4891#issuecomment-475924128 I have a suggestion. To add it to your repository execute a command: ```bash curl https://termbin.com/wymk | git am ``` Can you also add unit tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels
mik-laj commented on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels URL: https://github.com/apache/airflow/pull/4891#issuecomment-475924128 I have a suggestion. To add it to your repository execute a command: ```bash curl https://termbin.com/wymk | git am ``` Can you also add automatic tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] cchung100m commented on issue #4962: [AIRFLOW-3988] Airflow CLI - Sort commands alphabetically
cchung100m commented on issue #4962: [AIRFLOW-3988] Airflow CLI - Sort commands alphabetically URL: https://github.com/apache/airflow/pull/4962#issuecomment-475923954 Hi @ashb @XD-DENG Sorry for any inconvenience and thanks for the suggestions :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4963: [AIRFLOW-2227] Add delete method to Variable class
codecov-io edited a comment on issue #4963: [AIRFLOW-2227] Add delete method to Variable class URL: https://github.com/apache/airflow/pull/4963#issuecomment-475922883 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=h1) Report > Merging [#4963](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/40b7d2b5b61f3a98933f7af2ff4d651c487c50e3?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4963 +/- ## == + Coverage 75.74% 75.74% +<.01% == Files 458 458 Lines 2985029853 +3 == + Hits2260922612 +3 Misses 7241 7241 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4963/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.97% <100%> (-0.04%)` | :arrow_down: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4963/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.94% <100%> (+0.01%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=footer). Last update [40b7d2b...8ff9150](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4963: [AIRFLOW-2227] Add delete method to Variable class
codecov-io commented on issue #4963: [AIRFLOW-2227] Add delete method to Variable class URL: https://github.com/apache/airflow/pull/4963#issuecomment-475922883 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=h1) Report > Merging [#4963](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/40b7d2b5b61f3a98933f7af2ff4d651c487c50e3?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4963 +/- ## == + Coverage 75.74% 75.74% +<.01% == Files 458 458 Lines 2985029853 +3 == + Hits2260922612 +3 Misses 7241 7241 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4963/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.97% <100%> (-0.04%)` | :arrow_down: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4963/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.94% <100%> (+0.01%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=footer). Last update [40b7d2b...8ff9150](https://codecov.io/gh/apache/airflow/pull/4963?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
mik-laj commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#issuecomment-475922054 @Fokko @ashb I looked at SQL Alchemy documentation https://docs.sqlalchemy.org/en/latest/changelog/ and I liked the structure of this documentation. What do you think about spliting updating.rst and changelog.rst to multiple files? Currently, the descriptions in the updating.md file are very short and may not be enough. If we divide the file into parts, we will be able to add much better descriptions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
mik-laj commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#discussion_r268415837 ## File path: CONTRIBUTING.rst ## @@ -0,0 +1,479 @@ +.. Licensed to the Apache Software Foundation (ASF) under one Review comment: I also do not want two files. I forgot to delete the old file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4145) Allow RBAC roles permissions , VM to be overridable
[ https://issues.apache.org/jira/browse/AIRFLOW-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799876#comment-16799876 ] ASF GitHub Bot commented on AIRFLOW-4145: - XD-DENG commented on pull request #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable URL: https://github.com/apache/airflow/pull/4960 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow RBAC roles permissions , VM to be overridable > --- > > Key: AIRFLOW-4145 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4145 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4145) Allow RBAC roles permissions , VM to be overridable
[ https://issues.apache.org/jira/browse/AIRFLOW-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799877#comment-16799877 ] ASF subversion and git services commented on AIRFLOW-4145: -- Commit 40b7d2b5b61f3a98933f7af2ff4d651c487c50e3 in airflow's branch refs/heads/master from Tao Feng [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=40b7d2b ] [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable (#4960) > Allow RBAC roles permissions , VM to be overridable > --- > > Key: AIRFLOW-4145 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4145 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] mik-laj commented on a change in pull request #4842: [AIRFLOW-4014] Change DatastoreHook and add tests
mik-laj commented on a change in pull request #4842: [AIRFLOW-4014] Change DatastoreHook and add tests URL: https://github.com/apache/airflow/pull/4842#discussion_r268415354 ## File path: airflow/contrib/hooks/datastore_hook.py ## @@ -19,58 +19,73 @@ # import time + from googleapiclient.discovery import build + from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook class DatastoreHook(GoogleCloudBaseHook): """ -Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform -connection. +Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform connection. This object is not threads safe. If you want to make multiple requests simultaneously, you will need to create a hook per thread. + +:param version: The version of the API it is going to connect to. +:type version: str """ -def __init__(self, - datastore_conn_id='google_cloud_datastore_default', - delegate_to=None): +def __init__(self, datastore_conn_id='google_cloud_datastore_default', delegate_to=None, version='v1'): Review comment: Other hooks use order `api_version`, `gcp_conn_id`, `delegate_to`. They also use the name of the `api_version` instead name of `version`. I plan to standardize parameter names in operators and hooks in the future. The task is waiting in the queue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG merged pull request #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable
XD-DENG merged pull request #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable URL: https://github.com/apache/airflow/pull/4960 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference
mik-laj commented on issue #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference URL: https://github.com/apache/airflow/pull/4788#issuecomment-475919502 @Fokko i updated a PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference
mik-laj commented on a change in pull request #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference URL: https://github.com/apache/airflow/pull/4788#discussion_r268415175 ## File path: docs/build.sh ## @@ -23,7 +23,8 @@ set -e FWDIR="$(cd "`dirname "$0"`"; pwd)" cd "$FWDIR" -[ -d _build ] && rm -r _build +[[ -d "_build" ]] && rm -r _build +[[ -d "_autoapi" ]] && rm -r _autoapi Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dchaplinsky commented on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels
dchaplinsky commented on issue #4891: [AIRFLOW-4067] Telegram hook/operator to post messages to telegram channels URL: https://github.com/apache/airflow/pull/4891#issuecomment-475919162 @mik-laj can you check it now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference
mik-laj commented on a change in pull request #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference URL: https://github.com/apache/airflow/pull/4788#discussion_r268415031 ## File path: docs/autoapi_templates/index.rst ## @@ -0,0 +1,130 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +..http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +API Reference += + +Operators +- +Operators allow for generation of certain types of tasks that become nodes in +the DAG when instantiated. All operators derive from :class:`~airflow.models.BaseOperator` and +inherit many attributes and methods that way. + +There are 3 main types of operators: + +- Operators that performs an **action**, or tell another system to + perform an action +- **Transfer** operators move data from one system to another +- **Sensors** are a certain type of operator that will keep running until a + certain criterion is met. Examples include a specific file landing in HDFS or + S3, a partition appearing in Hive, or a specific time of the day. Sensors + are derived from :class:`~airflow.sensors.base_sensor_operator.BaseSensorOperator` and run a poke + method at a specified :attr:`~airflow.sensors.base_sensor_operator.BaseSensorOperator.poke_interval` until it returns ``True``. + +BaseOperator + +All operators are derived from :class:`~airflow.models.BaseOperator` and acquire much +functionality through inheritance. Since this is the core of the engine, +it's worth taking the time to understand the parameters of :class:`~airflow.models.BaseOperator` +to understand the primitive features that can be leveraged in your +DAGs. + +BaseSensorOperator +'' +All sensors are derived from :class:`~airflow.sensors.base_sensor_operator.BaseSensorOperator`. All sensors inherit +the :attr:`~airflow.sensors.base_sensor_operator.BaseSensorOperator.timeout` and :attr:`~airflow.sensors.base_sensor_operator.BaseSensorOperator.poke_interval` on top of the :class:`~airflow.models.BaseOperator` +attributes. + +Operators packages +'' +All operators are in the following packages: + +.. toctree:: + :includehidden: + :glob: + :maxdepth: 1 + + airflow/operators/index + + airflow/sensors/index + + airflow/contrib/operators/index + + airflow/contrib/sensors/index + + +Hooks +- +Hooks are interfaces to external platforms and databases, implementing a common +interface when possible and acting as building blocks for operators. All hooks +are derived from :class:`~airflow.hooks.base_hook.BaseHook`. + +Hooks packages +'' +All hooks are in the following packages: + +.. toctree:: + :includehidden: + :glob: + :maxdepth: 1 + + airflow/hooks/index + + airflow/contrib/hooks/index + + +Executors +- +Executors are the mechanism by which task instances get run. All executors are +derived from :class:`~airflow.executors.base_executor.BaseExecutor`. + +Executors packages +'' +All executors are in the following packages: + +.. toctree:: + :includehidden: + :glob: + :maxdepth: 1 + + airflow/executors/index + + airflow/contrib/executors/index + + +Models +-- + +Models are built on top of the SQLAlchemy ORM Base class, and instances are +persisted in the database. + +.. toctree:: + :includehidden: + :glob: + :maxdepth: 1 + + airflow/models/index + + +Core and community package Review comment: I added this fragment in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook
mik-laj commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook URL: https://github.com/apache/airflow/pull/4790#discussion_r268414576 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -27,6 +27,8 @@ import os import tempfile +from oauth2client.client import GOOGLE_APPLICATION_CREDENTIALS Review comment: I updated a PR. In addition, I added a new point to the list of changes: > some tests were dependent on the database. I added a mock. In the future I will try to fix the constructor. It should not connect to the database This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
codecov-io edited a comment on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#issuecomment-468903854 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=h1) Report > Merging [#4818](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cb32b7770d2000de8175c7df97bcbce4424890f9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4818 +/- ## == - Coverage 75.74% 75.74% -0.01% == Files 458 458 Lines 2985029849 -1 == - Hits2260922608 -1 Misses 7241 7241 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/bigquery\_sensor.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYmlncXVlcnlfc2Vuc29yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [...rflow/contrib/operators/bigquery\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9jaGVja19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/bigquery\_hook.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2JpZ3F1ZXJ5X2hvb2sucHk=) | `60.89% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_get\_data.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9nZXRfZGF0YS5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9vcGVyYXRvci5weQ==) | `93.57% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV90b19iaWdxdWVyeS5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [...ontrib/operators/bigquery\_table\_delete\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV90YWJsZV9kZWxldGVfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/gcs\_to\_s3.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fczMucHk=) | `96.77% <ø> (ø)` | :arrow_up: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `90.19% <ø> (-0.1%)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=footer). Last update [cb32b77...f1330ed](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
codecov-io edited a comment on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#issuecomment-468903854 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=h1) Report > Merging [#4818](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cb32b7770d2000de8175c7df97bcbce4424890f9?src=pr=desc) will **decrease** coverage by `0.65%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4818 +/- ## == - Coverage 75.74% 75.09% -0.66% == Files 458 458 Lines 2985029849 -1 == - Hits2260922414 -195 - Misses 7241 7435 +194 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/bigquery\_sensor.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYmlncXVlcnlfc2Vuc29yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [...rflow/contrib/operators/bigquery\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9jaGVja19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/bigquery\_hook.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2JpZ3F1ZXJ5X2hvb2sucHk=) | `60.89% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_get\_data.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9nZXRfZGF0YS5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9vcGVyYXRvci5weQ==) | `93.57% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV90b19iaWdxdWVyeS5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [...ontrib/operators/bigquery\_table\_delete\_operator.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV90YWJsZV9kZWxldGVfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/gcs\_to\_s3.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fczMucHk=) | `96.77% <ø> (ø)` | :arrow_up: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `90.19% <ø> (-0.1%)` | :arrow_down: | | ... and [14 more](https://codecov.io/gh/apache/airflow/pull/4818/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=footer). Last update [cb32b77...f1330ed](https://codecov.io/gh/apache/airflow/pull/4818?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ryanyuan commented on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
ryanyuan commented on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#issuecomment-475916500 @potiuk @kaxil Thanks for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3659) Create Google Cloud Transfer Service Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799840#comment-16799840 ] ASF GitHub Bot commented on AIRFLOW-3659: - kaxil commented on pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create Google Cloud Transfer Service Operators > -- > > Key: AIRFLOW-3659 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3659 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Jarek Potiuk >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] Fokko commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
Fokko commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951#discussion_r268412322 ## File path: airflow/models/__init__.py ## @@ -3515,6 +3519,7 @@ def get_template_env(self): env = jinja2.Environment( loader=jinja2.FileSystemLoader(searchpath), +undefined=self.template_undefined, Review comment: No problem, any time @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TheOriginalAlex opened a new pull request #4964: [AIRFLOW-XXX]
TheOriginalAlex opened a new pull request #4964: [AIRFLOW-XXX]
URL: https://github.com/apache/airflow/pull/4964
Add to the list of engineers working on Airflow at Bombora Inc
Signed-off-by: Alex Johnson
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Added the names of a few engineers working with Airflow in Bombora to the
README
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just a change to the README
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook
kaxil commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook URL: https://github.com/apache/airflow/pull/4790#discussion_r268412275 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -27,6 +27,8 @@ import os import tempfile +from oauth2client.client import GOOGLE_APPLICATION_CREDENTIALS Review comment: Can we use https://google-auth.readthedocs.io/en/latest/ instead? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook
kaxil commented on a change in pull request #4790: [AIRFLOW-3968][2/3] Refactor base GCP hook URL: https://github.com/apache/airflow/pull/4790#discussion_r268412252 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -27,6 +27,8 @@ import os import tempfile +from oauth2client.client import GOOGLE_APPLICATION_CREDENTIALS Review comment: isn't oauth2client library deprecated? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3659) Create Google Cloud Transfer Service Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799841#comment-16799841 ] ASF subversion and git services commented on AIRFLOW-3659: -- Commit 4c6a591a90dca178557a2c63ffae4e4703d9eec7 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4c6a591 ] [AIRFLOW-3659] Create Google Cloud Transfer Service Operators (#4792) Co-authored-by: Antoni Smolinski > Create Google Cloud Transfer Service Operators > -- > > Key: AIRFLOW-3659 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3659 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Jarek Potiuk >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3659) Create Google Cloud Transfer Service Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3659. - Resolution: Fixed Fix Version/s: 2.0.0 > Create Google Cloud Transfer Service Operators > -- > > Key: AIRFLOW-3659 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3659 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Jarek Potiuk >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] kaxil merged pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators
kaxil merged pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #4749: [AIRFLOW-3934] Increase standard Dataproc PD size
kaxil commented on issue #4749: [AIRFLOW-3934] Increase standard Dataproc PD size URL: https://github.com/apache/airflow/pull/4749#issuecomment-475911484 @Fokko Let's make this change but also add a note in `Updating.md` that the default value has changed. I am sure 99.99% users don't use this default value but better be explicit than sorry This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's
codecov-io commented on issue #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's URL: https://github.com/apache/airflow/pull/4808#issuecomment-475911089 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=h1) Report > Merging [#4808](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e220ac5bcfd56f048f552f47e7e1c5813f7b928f?src=pr=desc) will **increase** coverage by `1.45%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4808 +/- ## == + Coverage75.3% 76.75% +1.45% == Files 450 456 +6 Lines 2902334723+5700 == + Hits2185526651+4796 - Misses 7168 8072 +904 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `79.78% <ø> (+2.92%)` | :arrow_up: | | [airflow/example\_dags/tutorial.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvdHV0b3JpYWwucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `94.7% <100%> (+1.79%)` | :arrow_up: | | [airflow/contrib/hooks/winrm\_hook.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3dpbnJtX2hvb2sucHk=) | `79.31% <0%> (-20.69%)` | :arrow_down: | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `81.25% <0%> (-14.75%)` | :arrow_down: | | [.../kubernetes\_request\_factory/pod\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkvcG9kX3JlcXVlc3RfZmFjdG9yeS5weQ==) | `86.56% <0%> (-13.44%)` | :arrow_down: | | [airflow/contrib/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZV9jbGllbnQucHk=) | `62.06% <0%> (-12.94%)` | :arrow_down: | | [airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5) | `78.28% <0%> (-11.88%)` | :arrow_down: | | [airflow/contrib/operators/mysql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9teXNxbF90b19nY3MucHk=) | `80.2% <0%> (-9.87%)` | :arrow_down: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `84.48% <0%> (-7.19%)` | :arrow_down: | | ... and [91 more](https://codecov.io/gh/apache/airflow/pull/4808/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=footer). Last update [e220ac5...4e29315](https://codecov.io/gh/apache/airflow/pull/4808?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #4897: [AIRFLOW-4062] Clear install extra package command
kaxil commented on issue #4897: [AIRFLOW-4062] Clear install extra package command URL: https://github.com/apache/airflow/pull/4897#issuecomment-475910859 I agree with Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] vsoch commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow
vsoch commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846#issuecomment-475910707 wow what a beautiful integration! I didn't know codecov had come so far (I remember it just spitting out a txt report after testing). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #4925: [AIRFLOW-4103] allow uppercase letters in dataflow job names
kaxil commented on issue #4925: [AIRFLOW-4103] allow uppercase letters in dataflow job names URL: https://github.com/apache/airflow/pull/4925#issuecomment-475910545 I agree with your opinion. Can you add documentation (maybe in docstring) or a note somewhere mentioning that all uppercase characters would be changed to lowercase This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951#discussion_r268411642 ## File path: airflow/models/__init__.py ## @@ -3515,6 +3519,7 @@ def get_template_env(self): env = jinja2.Environment( loader=jinja2.FileSystemLoader(searchpath), +undefined=self.template_undefined, Review comment: Oh yes, you are right... my bad :) Thanks @Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
kaxil commented on a change in pull request #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#discussion_r268411591 ## File path: UPDATING.md ## @@ -24,6 +24,15 @@ assists users migrating to a new version. ## Airflow Master + Unify default conn_id for Google Cloud Platform + +Previously not all hooks and operators related to Google Cloud Platform use +``google_cloud_default`` as a default conn_id. There is currently one default +variant. Values like ``google_cloud_storage_default``, ``bigquery_default``, +``google_cloud_datastore_default`` have been abandoned. The configuration of +connections in the database have been preserved, but in every place where they +were used, they should be indicated explicitly. Review comment: Can we rephrase the last line ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
kaxil commented on a change in pull request #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#discussion_r268411479 ## File path: UPDATING.md ## @@ -24,6 +24,15 @@ assists users migrating to a new version. ## Airflow Master + Unify default conn_id for Google Cloud Platform + +Previously not all hooks and operators related to Google Cloud Platform use +``google_cloud_default`` as a default conn_id. There is currently one default +variant. Values like ``google_cloud_storage_default``, ``bigquery_default``, +``google_cloud_datastore_default`` have been abandoned. The configuration of Review comment: ```suggestion ``google_cloud_datastore_default`` have been deprecated. The configuration of ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
Fokko commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951#discussion_r268411345 ## File path: airflow/models/__init__.py ## @@ -3515,6 +3519,7 @@ def get_template_env(self): env = jinja2.Environment( loader=jinja2.FileSystemLoader(searchpath), +undefined=self.template_undefined, Review comment: The default is Undefined: https://github.com/pallets/jinja/blob/master/jinja2/environment.py#L279 This is still the case if it isn't passed explicitly: https://github.com/apache/airflow/pull/4951/files/65af90c017022c2fb31b24d336e0de45934120cd#diff-664e0133fa172b6e0060f2dd7b2c3504R2990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4939: [AIRFLOW-4120] Modify SchedulerJob.manage_slas to respect zero timedelta SLAs
Fokko commented on issue #4939: [AIRFLOW-4120] Modify SchedulerJob.manage_slas to respect zero timedelta SLAs URL: https://github.com/apache/airflow/pull/4939#issuecomment-475909464 @cooltoast would it be possible to provide a test? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
Fokko commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME URL: https://github.com/apache/airflow/pull/4705#discussion_r268411218 ## File path: airflow/__init__.py ## @@ -37,9 +37,6 @@ from airflow.models import DAG from airflow.exceptions import AirflowException -if settings.DAGS_FOLDER not in sys.path: Review comment: For me, appending to `sys.path` is a code smell imo, I think it is good to just remove it :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #4963: [AIRFLOW-2227] Add delete method to Variable class
Fokko commented on a change in pull request #4963: [AIRFLOW-2227] Add delete method to Variable class URL: https://github.com/apache/airflow/pull/4963#discussion_r268411169 ## File path: airflow/bin/cli.py ## @@ -343,7 +343,7 @@ def variables(args): print(e) if args.delete: with db.create_session() as session: Review comment: We can remove the create_session here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4571: [AIRFLOW-3419] S3_hook.select_key is broken on Python3
Fokko commented on issue #4571: [AIRFLOW-3419] S3_hook.select_key is broken on Python3 URL: https://github.com/apache/airflow/pull/4571#issuecomment-475909145 Restarted Travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow
codecov-io commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846#issuecomment-475908770 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=h1) Report > Merging [#4846](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7288ad3b5305031bbb526b9b55a20510579c6b2e?src=pr=desc) will **increase** coverage by `0.66%`. > The diff coverage is `70.7%`. [](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4846 +/- ## == + Coverage 75.45% 76.12% +0.66% == Files 451 457 +6 Lines 2909931219+2120 == + Hits2195723765+1808 - Misses 7142 7454 +312 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.29% <0%> (-0.06%)` | :arrow_down: | | [...ntrib/example\_dags/example\_singularity\_operator.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3Npbmd1bGFyaXR5X29wZXJhdG9yLnB5) | `0% <0%> (ø)` | | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `94.4% <100%> (ø)` | :arrow_up: | | [airflow/macros/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9tYWNyb3MvX19pbml0X18ucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.46% <100%> (+3.84%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `94.87% <100%> (+1.95%)` | :arrow_up: | | [airflow/contrib/operators/singularity\_operator.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zaW5ndWxhcml0eV9vcGVyYXRvci5weQ==) | `80.51% <80.51%> (ø)` | | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `86.02% <0%> (-12.65%)` | :arrow_down: | | ... and [65 more](https://codecov.io/gh/apache/airflow/pull/4846/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=footer). Last update [7288ad3...95a7084](https://codecov.io/gh/apache/airflow/pull/4846?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference
Fokko commented on issue #4788: [AIRFLOW-3811][3/3] Add automatic generation of API Reference URL: https://github.com/apache/airflow/pull/4788#issuecomment-475908606 Looking good @mik-laj Can you fix the merge conflict? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3541) Include --useAvroLogicalTypes setting for the Airflow operator GoogleCloudStorageToBigQueryOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799816#comment-16799816
]
ASF subversion and git services commented on AIRFLOW-3541:
--
Commit 2f6bcc9f350174e295426fd41f14a8e72bc1b75b in airflow's branch
refs/heads/v1-10-test from BrechtDeVlieger
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2f6bcc9 ]
[AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
> Include --useAvroLogicalTypes setting for the Airflow operator
> GoogleCloudStorageToBigQueryOperator
> ---
>
> Key: AIRFLOW-3541
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Environment: Composer
>Reporter: Monica Parpi
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.10.3
>
>
> Getting an error [1] when loading data from GCS into BigQuery using the
> GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
> {
> "name": "date_requested",
> "type": "int",
> "logicalType": "date",
> }
> This column contains the number of days since the epoch as indicated here [2].
> It is possible to load this data into a DATE column by using the
> "--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
> It would be very useful if this setting could be enabled via the Airflow
> operator that performs the equivalent data load [4].
> Thank you!
> [ [1]
> https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
>
> [[2]
> http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
> [3]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
> [4]
> [https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] Fokko commented on issue #4957: [AIRFLOW-161] New redirect route and extra links
Fokko commented on issue #4957: [AIRFLOW-161] New redirect route and extra links URL: https://github.com/apache/airflow/pull/4957#issuecomment-475908360 It looks like we have an unhappy test: ``` == 47) ERROR: test_extra_links_error_raised (tests.www.test_views.TestExtraLinks) -- Traceback (most recent call last): tests/www/test_views.py line 1633 in setUp self.app = application.create_app().test_client() AttributeError: 'tuple' object has no attribute 'test_client' ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951#discussion_r268410812 ## File path: airflow/models/__init__.py ## @@ -3515,6 +3519,7 @@ def get_template_env(self): env = jinja2.Environment( loader=jinja2.FileSystemLoader(searchpath), +undefined=self.template_undefined, Review comment: Please correct me if I am wrong, this is a breaking change, isn't it ?? Behavior before this PR is if the variable is not defined it would error. But after merging this PR: it would not raise an error. @jmcarp @Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
kaxil commented on a change in pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951#discussion_r268410812 ## File path: airflow/models/__init__.py ## @@ -3515,6 +3519,7 @@ def get_template_env(self): env = jinja2.Environment( loader=jinja2.FileSystemLoader(searchpath), +undefined=self.template_undefined, Review comment: Please correct me if I am wrong, this is a breaking change, isn't it ?? Behavior before this PR is if the variable is not defined it would error. But after merging this PR: it would not raise an error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3541) Include --useAvroLogicalTypes setting for the Airflow operator GoogleCloudStorageToBigQueryOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-3541.
---
Resolution: Fixed
Fix Version/s: 1.10.3
2.0.0
> Include --useAvroLogicalTypes setting for the Airflow operator
> GoogleCloudStorageToBigQueryOperator
> ---
>
> Key: AIRFLOW-3541
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Environment: Composer
>Reporter: Monica Parpi
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.10.3
>
>
> Getting an error [1] when loading data from GCS into BigQuery using the
> GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
> {
> "name": "date_requested",
> "type": "int",
> "logicalType": "date",
> }
> This column contains the number of days since the epoch as indicated here [2].
> It is possible to load this data into a DATE column by using the
> "--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
> It would be very useful if this setting could be enabled via the Airflow
> operator that performs the equivalent data load [4].
> Thank you!
> [ [1]
> https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
>
> [[2]
> http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
> [3]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
> [4]
> [https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3541) Include --useAvroLogicalTypes setting for the Airflow operator GoogleCloudStorageToBigQueryOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799815#comment-16799815
]
ASF subversion and git services commented on AIRFLOW-3541:
--
Commit fe8c9a513d468d230b5cca77535d60f18295f05d in airflow's branch
refs/heads/master from BrechtDeVlieger
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fe8c9a5 ]
[AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
> Include --useAvroLogicalTypes setting for the Airflow operator
> GoogleCloudStorageToBigQueryOperator
> ---
>
> Key: AIRFLOW-3541
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Environment: Composer
>Reporter: Monica Parpi
>Priority: Major
> Labels: pull-request-available
>
> Getting an error [1] when loading data from GCS into BigQuery using the
> GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
> {
> "name": "date_requested",
> "type": "int",
> "logicalType": "date",
> }
> This column contains the number of days since the epoch as indicated here [2].
> It is possible to load this data into a DATE column by using the
> "--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
> It would be very useful if this setting could be enabled via the Airflow
> operator that performs the equivalent data load [4].
> Thank you!
> [ [1]
> https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
>
> [[2]
> http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
> [3]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
> [4]
> [https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3541) Include --useAvroLogicalTypes setting for the Airflow operator GoogleCloudStorageToBigQueryOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799814#comment-16799814
]
ASF GitHub Bot commented on AIRFLOW-3541:
-
Fokko commented on pull request #4553: [AIRFLOW-3541] Add Avro logical type
conversion to bigquery hook
URL: https://github.com/apache/airflow/pull/4553
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Include --useAvroLogicalTypes setting for the Airflow operator
> GoogleCloudStorageToBigQueryOperator
> ---
>
> Key: AIRFLOW-3541
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Environment: Composer
>Reporter: Monica Parpi
>Priority: Major
> Labels: pull-request-available
>
> Getting an error [1] when loading data from GCS into BigQuery using the
> GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
> {
> "name": "date_requested",
> "type": "int",
> "logicalType": "date",
> }
> This column contains the number of days since the epoch as indicated here [2].
> It is possible to load this data into a DATE column by using the
> "--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
> It would be very useful if this setting could be enabled via the Airflow
> operator that performs the equivalent data load [4].
> Thank you!
> [ [1]
> https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
>
> [[2]
> http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
> [3]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
> [4]
> [https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] Fokko merged pull request #4553: [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook
Fokko merged pull request #4553: [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook URL: https://github.com/apache/airflow/pull/4553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4553: [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook
Fokko commented on issue #4553: [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook URL: https://github.com/apache/airflow/pull/4553#issuecomment-475907812 As a committer on Avro as well, I think I can chip in here. I would say that having BQ to pick up the logical types is a behavior that we would like to use. For example, the physical type of a TIMESTAMP is an int (seconds since epoch), but we would still like to have BQ interpret this a timestamp instead of a plain integer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify
different methods of working out AIRFLOW_HOME
URL: https://github.com/apache/airflow/pull/4705#discussion_r268410507
##
File path: airflow/lineage/__init__.py
##
@@ -36,7 +36,6 @@ def _get_backend():
try:
_backend_str = conf.get("lineage", "backend")
-prepare_classpath()
Review comment:
Just want to know why was it present if we can just remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME URL: https://github.com/apache/airflow/pull/4705#discussion_r268410216 ## File path: UPDATING.md ## @@ -219,6 +219,19 @@ if foo is None: This changes the behaviour if you previously explicitly provided `None` as a default value. If your code expects a `KeyError` to be thrown, then don't pass the `default_var` argument. +### Removal of `airflow_home` config setting + +There were previously two ways of specifying the Airflow "home" directory +(`~/airflow` by default): the `AIRFLOW_HOME` environment variable, and the +`airflow_home` config setting in the `[core]` section. + +If they had two different values different parts of the code base would end up +with different values. The config setting has been deprecated, and you should +remove the value form the config files and set `AIRFLOW_HOME` environment Review comment: Typo: ```suggestion remove the value from the config file and set `AIRFLOW_HOME` environment ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4939: [AIRFLOW-4120] Modify SchedulerJob.manage_slas to respect zero timedelta SLAs
codecov-io commented on issue #4939: [AIRFLOW-4120] Modify SchedulerJob.manage_slas to respect zero timedelta SLAs URL: https://github.com/apache/airflow/pull/4939#issuecomment-475907566 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=h1) Report > Merging [#4939](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/dce92a54190155898c75c0f3392d42fb28f1884a?src=pr=desc) will **increase** coverage by `0.44%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4939 +/- ## == + Coverage 75.59% 76.03% +0.44% == Files 454 455 +1 Lines 2919731052+1855 == + Hits2207123610+1539 - Misses 7126 7442 +316 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `77% <50%> (+0.12%)` | :arrow_up: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `86.02% <0%> (-12.59%)` | :arrow_down: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `84.48% <0%> (-7.19%)` | :arrow_down: | | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `84.1% <0%> (-2.69%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.51% <0%> (-0.36%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_vision\_operator.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdmlzaW9uX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/example\_dags/example\_gcp\_vision.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcF92aXNpb24ucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_glue\_catalog\_hook.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19nbHVlX2NhdGFsb2dfaG9vay5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/stats.py](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree#diff-YWlyZmxvdy9zdGF0cy5weQ==) | `68.18% <0%> (ø)` | | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/4939/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=footer). Last update [dce92a5...98333ab](https://codecov.io/gh/apache/airflow/pull/4939?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify
different methods of working out AIRFLOW_HOME
URL: https://github.com/apache/airflow/pull/4705#discussion_r268410535
##
File path: airflow/plugins_manager.py
##
@@ -122,10 +122,11 @@ def is_valid_plugin(plugin_obj, existing_plugins):
return False
-plugins_folder = configuration.conf.get('core', 'plugins_folder')
-if not plugins_folder:
-plugins_folder = configuration.conf.get('core', 'airflow_home') +
'/plugins'
-plugins_folder = os.path.expanduser(plugins_folder)
+plugins_folder = configuration.conf.get(
Review comment:
Can we not add this line to settings.py and just use
`plugins_folder=settings.plugins_folder` for uniformity?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify
different methods of working out AIRFLOW_HOME
URL: https://github.com/apache/airflow/pull/4705#discussion_r268410453
##
File path: airflow/configuration.py
##
@@ -544,6 +544,26 @@ def parameterized_config(template):
conf.read(AIRFLOW_CONFIG)
+if conf.has_option('core', 'AIRFLOW_HOME'):
Review comment:
I am not able to understand how we get this? I understand `conf.get('core',
'airflow_home')` but do we get this from directly environment variable? If so,
how is it an option in `core` section..
Shouldn't it be just
```
if 'AIRFLOW_HOME' in os.environ and conf.get('core', 'airflow_home'):
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME
kaxil commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME URL: https://github.com/apache/airflow/pull/4705#discussion_r268410311 ## File path: airflow/__init__.py ## @@ -37,9 +37,6 @@ from airflow.models import DAG from airflow.exceptions import AirflowException -if settings.DAGS_FOLDER not in sys.path: Review comment: Why are we removing this from here?? I see that we have the same logic in `airflow/settings.py` but have not investigated why! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-2961) Speed up test_backfill_examples test
[ https://issues.apache.org/jira/browse/AIRFLOW-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-2961: - Assignee: Fokko Driesprong > Speed up test_backfill_examples test > > > Key: AIRFLOW-2961 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2961 > Project: Apache Airflow > Issue Type: Bug >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4131) Allow users to control undefined behavior in templates
[ https://issues.apache.org/jira/browse/AIRFLOW-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799810#comment-16799810 ] ASF GitHub Bot commented on AIRFLOW-4131: - Fokko commented on pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow users to control undefined behavior in templates > -- > > Key: AIRFLOW-4131 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4131 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Fix For: 2.0.0 > > > As a user, I want to configure templates to raise exceptions on undefined > variables rather than silently replacing them with the empty string. I > propose adding a `template_undefined` option to `DAG` to support this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] Fokko commented on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs
Fokko commented on issue #4818: [AIRFLOW-3987] Unify GCP's Connection IDs URL: https://github.com/apache/airflow/pull/4818#issuecomment-475907173 @ryanyuan Could you resolve the conflicts? :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] vsoch commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow
vsoch commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846#issuecomment-475906977 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow
Fokko commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846#issuecomment-475906912 I've restarted the test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #4756: [AIRFLOW-3677] Improve check operator module test coverage
feluelle commented on a change in pull request #4756: [AIRFLOW-3677] Improve
check operator module test coverage
URL: https://github.com/apache/airflow/pull/4756#discussion_r268410371
##
File path: airflow/operators/check_operator.py
##
@@ -215,8 +223,7 @@ def __init__(
self.days_back = -abs(days_back)
self.conn_id = conn_id
sqlexp = ', '.join(self.metrics_sorted)
-sqlt = ("SELECT {sqlexp} FROM {table}"
-" WHERE {date_filter_column}=").format(**locals())
+sqlt = "SELECT {sqlexp} FROM {table} WHERE
{date_filter_column}=".format(**locals())
Review comment:
Sure. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-4131) Allow users to control undefined behavior in templates
[ https://issues.apache.org/jira/browse/AIRFLOW-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799811#comment-16799811 ] ASF subversion and git services commented on AIRFLOW-4131: -- Commit 567c4f4e8f70fd27e5850a9331011412ba5a3a21 in airflow's branch refs/heads/master from Joshua Carp [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=567c4f4 ] [AIRFLOW-4131] Make template undefined behavior configurable. (#4951) > Allow users to control undefined behavior in templates > -- > > Key: AIRFLOW-4131 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4131 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Fix For: 2.0.0 > > > As a user, I want to configure templates to raise exceptions on undefined > variables rather than silently replacing them with the empty string. I > propose adding a `template_undefined` option to `DAG` to support this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4131) Allow users to control undefined behavior in templates
[ https://issues.apache.org/jira/browse/AIRFLOW-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4131. --- Resolution: Fixed Fix Version/s: 2.0.0 > Allow users to control undefined behavior in templates > -- > > Key: AIRFLOW-4131 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4131 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Priority: Trivial > Fix For: 2.0.0 > > > As a user, I want to configure templates to raise exceptions on undefined > variables rather than silently replacing them with the empty string. I > propose adding a `template_undefined` option to `DAG` to support this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4131) Allow users to control undefined behavior in templates
[ https://issues.apache.org/jira/browse/AIRFLOW-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-4131: - Assignee: Josh Carp > Allow users to control undefined behavior in templates > -- > > Key: AIRFLOW-4131 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4131 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Fix For: 2.0.0 > > > As a user, I want to configure templates to raise exceptions on undefined > variables rather than silently replacing them with the empty string. I > propose adding a `template_undefined` option to `DAG` to support this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] Fokko merged pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable.
Fokko merged pull request #4951: [AIRFLOW-4131] Make template undefined behavior configurable. URL: https://github.com/apache/airflow/pull/4951 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4112) Remove beeline_default as default connection
[ https://issues.apache.org/jira/browse/AIRFLOW-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799805#comment-16799805 ] ASF GitHub Bot commented on AIRFLOW-4112: - Fokko commented on pull request #4934: [AIRFLOW-4112] Remove beeline_default in default connection URL: https://github.com/apache/airflow/pull/4934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove beeline_default as default connection > > > Key: AIRFLOW-4112 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4112 > Project: Apache Airflow > Issue Type: Improvement > Components: db >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: pull-request-available > Fix For: 1.10.3 > > > We use both `beeline_default` and `hive_cli_default` as our default when we > run `airflow initdb`. But in airflow source part we only use > `hive_cli_default` as conn_id in hive related hook/operator and only use > `beeline_default` in airflow test part for test hive hook/operator. That why > I think we should merge then as one default connection. > > And in [Hive > doc|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-DeprecationinfavorofBeelineCLI] > could know that hive cli will be deprecation in favor of Beeline, so in this > situation I think we should reomve 'beeline_default' and change > 'hive_cli_default' as same configure as 'beeline_default'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4112) Remove beeline_default as default connection
[ https://issues.apache.org/jira/browse/AIRFLOW-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799806#comment-16799806 ] ASF subversion and git services commented on AIRFLOW-4112: -- Commit 1bbc3b8c87c0704696c2886964d5666e5f3ac019 in airflow's branch refs/heads/master from Jiajie Zhong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1bbc3b8 ] [AIRFLOW-4112] Remove beeline_default in default connection (#4934) We use both `beeline_default` and `hive_cli_default` as our default when we run `airflow initdb`. But in airflow source folder we only use `hive_cli_default` as conn_id in hive related hook/operator and only use `beeline_default` in airflow test folder for test hive hook/operator. That why I think we should merge then as one default connection And in [Hive doc](https://cwiki.apache.org/confluence/ display/Hive/LanguageManual+Cli# LanguageManualCli-DeprecationinfavorofBeelineCLI) could know that hive cli will be deprecation in favor of Beeline. In this situation I think we should remove `beeline_default` and change `hive_cli_default` as same configure as `beeline_default` > Remove beeline_default as default connection > > > Key: AIRFLOW-4112 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4112 > Project: Apache Airflow > Issue Type: Improvement > Components: db >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: pull-request-available > Fix For: 1.10.3 > > > We use both `beeline_default` and `hive_cli_default` as our default when we > run `airflow initdb`. But in airflow source part we only use > `hive_cli_default` as conn_id in hive related hook/operator and only use > `beeline_default` in airflow test part for test hive hook/operator. That why > I think we should merge then as one default connection. > > And in [Hive > doc|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-DeprecationinfavorofBeelineCLI] > could know that hive cli will be deprecation in favor of Beeline, so in this > situation I think we should reomve 'beeline_default' and change > 'hive_cli_default' as same configure as 'beeline_default'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4112) Remove beeline_default as default connection
[ https://issues.apache.org/jira/browse/AIRFLOW-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4112. --- Resolution: Fixed Fix Version/s: (was: 1.10.3) 2.0.0 > Remove beeline_default as default connection > > > Key: AIRFLOW-4112 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4112 > Project: Apache Airflow > Issue Type: Improvement > Components: db >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > > We use both `beeline_default` and `hive_cli_default` as our default when we > run `airflow initdb`. But in airflow source part we only use > `hive_cli_default` as conn_id in hive related hook/operator and only use > `beeline_default` in airflow test part for test hive hook/operator. That why > I think we should merge then as one default connection. > > And in [Hive > doc|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-DeprecationinfavorofBeelineCLI] > could know that hive cli will be deprecation in favor of Beeline, so in this > situation I think we should reomve 'beeline_default' and change > 'hive_cli_default' as same configure as 'beeline_default'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] Fokko merged pull request #4934: [AIRFLOW-4112] Remove beeline_default in default connection
Fokko merged pull request #4934: [AIRFLOW-4112] Remove beeline_default in default connection URL: https://github.com/apache/airflow/pull/4934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmsanders commented on issue #4963: [AIRFLOW-2227] Add delete method to Variable class
jmsanders commented on issue #4963: [AIRFLOW-2227] Add delete method to Variable class URL: https://github.com/apache/airflow/pull/4963#issuecomment-475903994 [The tests appear to be passing locally.](https://travis-ci.com/jmsanders/airflow/builds/105538832) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #4756: [AIRFLOW-3677] Improve check operator module test coverage
Fokko commented on a change in pull request #4756: [AIRFLOW-3677] Improve check
operator module test coverage
URL: https://github.com/apache/airflow/pull/4756#discussion_r268409248
##
File path: airflow/operators/check_operator.py
##
@@ -215,8 +223,7 @@ def __init__(
self.days_back = -abs(days_back)
self.conn_id = conn_id
sqlexp = ', '.join(self.metrics_sorted)
-sqlt = ("SELECT {sqlexp} FROM {table}"
-" WHERE {date_filter_column}=").format(**locals())
+sqlt = "SELECT {sqlexp} FROM {table} WHERE
{date_filter_column}=".format(**locals())
Review comment:
Can we get rid of `**locals()` here and be explicit?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4861: [AIRFLOW-4034] Make string formatting explicit by removing **locals()
Fokko commented on issue #4861: [AIRFLOW-4034] Make string formatting explicit by removing **locals() URL: https://github.com/apache/airflow/pull/4861#issuecomment-475903687 Ok, let me know! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4905: [AIRFLOW-4072] enable GKEPodOperator xcom
Fokko commented on issue #4905: [AIRFLOW-4072] enable GKEPodOperator xcom URL: https://github.com/apache/airflow/pull/4905#issuecomment-475903420 Yes, this looks good. One minor thing, we need to set the `extract_xcom` to True: https://github.com/apache/airflow/blob/cf934be127206d2d75dce49f1fb0b2c2ffcdcb63/airflow/contrib/kubernetes/pod_launcher.py#L43 We should get rid of the above `extract_xcom` parameter, and always return the xcom. The BaseOperator will take care of xcom being enabled, or not: https://github.com/apache/airflow/blob/cf934be127206d2d75dce49f1fb0b2c2ffcdcb63/airflow/models/__init__.py#L1397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2227) Allow variable migrations by adding deletion
[
https://issues.apache.org/jira/browse/AIRFLOW-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799777#comment-16799777
]
ASF GitHub Bot commented on AIRFLOW-2227:
-
jmsanders commented on pull request #4963: [AIRFLOW-2227] Add delete method to
Variable class
URL: https://github.com/apache/airflow/pull/4963
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2227
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add a delete method to the Variable class that no-ops if the Variable
already does not exist. Additionally, refactor usages of
`session.query(Variable).filter(key).delete()` to use the new
`Variable.delete(key)` instead.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- `tests.core:CoreTest.test_variable_delete`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow variable migrations by adding deletion
>
>
> Key: AIRFLOW-2227
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2227
> Project: Apache Airflow
> Issue Type: Improvement
> Components: models
>Reporter: nadav
>Priority: Minor
>
> In order to allow using migration (via alembic or customized scripts) one
> need the ability to delete variables (delete function).
> Another utility function was added to allow retrieval of all the current
> variables keys.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] jmsanders opened a new pull request #4963: [AIRFLOW-2227] Add delete method to Variable class
jmsanders opened a new pull request #4963: [AIRFLOW-2227] Add delete method to
Variable class
URL: https://github.com/apache/airflow/pull/4963
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2227
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add a delete method to the Variable class that no-ops if the Variable
already does not exist. Additionally, refactor usages of
`session.query(Variable).filter(key).delete()` to use the new
`Variable.delete(key)` instead.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- `tests.core:CoreTest.test_variable_delete`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mans2singh edited a comment on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor
mans2singh edited a comment on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor URL: https://github.com/apache/airflow/pull/4887#issuecomment-475895414 @zhongjiajie - Thanks for your comments. I generally try to keep conditions separate so that they can log specific messages for easy debugging. I was also using a try block is so that I can log the exception and then propagate it. But I agree with your feedback and that it makes the code more concise and have made the recommended changes. I am not sure of what would be an intuitive name for the sqs sensor. I was thinking that `sensor` will indicate it subscribes to sqs. But if SQSPushXComSensor/SQSPushSensor or any other name is more expressive, please let me know. Thanks again for your time and advice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process
Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process URL: https://github.com/apache/airflow/pull/4896#issuecomment-475898449 https://travis-ci.org/Fokko/incubator-airflow/builds/510390240 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process
Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process URL: https://github.com/apache/airflow/pull/4896#issuecomment-475898406 @zhongjiajie Please `[ci skip]` from the commit message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process
Fokko commented on issue #4896: [AIRFLOW-4061] Remove incubator in CI process URL: https://github.com/apache/airflow/pull/4896#issuecomment-475898157 I think we should get this in for now. We should get rid of all incubator references. I've also requested to remove the old repository: https://jira.apache.org/jira/browse/INFRA-18084 Did the CI run with this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4897: [AIRFLOW-4062] Clear install extra package command
Fokko commented on issue #4897: [AIRFLOW-4062] Clear install extra package command URL: https://github.com/apache/airflow/pull/4897#issuecomment-475897856 I've seen people make this mistake often. I'll would like to get this in. @zhongjiajie. Can you rebase? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #4855: [AIRFLOW-2421] - HTTPHook set verify True by default
Fokko commented on issue #4855: [AIRFLOW-2421] - HTTPHook set verify True by default URL: https://github.com/apache/airflow/pull/4855#issuecomment-475897775 @OmerJog Can you rebase? I'll merge for Airflow 2.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
Fokko commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#discussion_r268406187 ## File path: CONTRIBUTING.rst ## @@ -0,0 +1,479 @@ +.. Licensed to the Apache Software Foundation (ASF) under one Review comment: This looks good. I think we should remove the old UPDATING. I agree with Ash that maintaining both of them isn't a good idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #4962: [AIRFLOW-3988] Airflow CLI - Sort commands alphabetically
ashb commented on issue #4962: [AIRFLOW-3988] Airflow CLI - Sort commands alphabetically URL: https://github.com/apache/airflow/pull/4962#issuecomment-475897298 You can make our lives as committers easier by actually filling out the pull request template This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
