[GitHub] [airflow-site] kgabryje opened a new pull request #154: fix not working links
kgabryje opened a new pull request #154: fix not working links URL: https://github.com/apache/airflow-site/pull/154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kgabryje opened a new pull request #155: removed additional accordion arrow on firefox
kgabryje opened a new pull request #155: removed additional accordion arrow on firefox URL: https://github.com/apache/airflow-site/pull/155 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] deshraj commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request
deshraj commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request URL: https://github.com/apache/airflow/pull/6380#issuecomment-553291432 @ashb @acroos is there a timeline for this fix so that it can be included in the next release? Would really appreciate if can be done sooner. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-3457) Refactor: Move User out of models.py
[ https://issues.apache.org/jira/browse/AIRFLOW-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971140#comment-16971140 ] David de L. edited comment on AIRFLOW-3457 at 11/13/19 8:39 AM: [~ash] [~basph] [~fokko] why was `user.py` [removed|https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/models/__init__.py#L38-L41] in `master`? It is still used by multiple auth backends on `master` (there's also no mention of this is `UPDATING.md`): [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/kerberos_auth.py#L48] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/github_enterprise_auth.py#L38] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/google_auth.py#L38] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/ldap_auth.py#L127] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/password_auth.py#L54] was (Author: ddelange): [~ash] why was `user.py` [removed|https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/models/__init__.py#L38-L41] in `master`? It is still used by multiple auth backends on `master` (there's also no mention of this is `UPDATING.md`): [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/kerberos_auth.py#L48] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/github_enterprise_auth.py#L38] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/google_auth.py#L38] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/ldap_auth.py#L127] [https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/contrib/auth/backends/password_auth.py#L54] > Refactor: Move User out of models.py > > > Key: AIRFLOW-3457 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3457 > Project: Apache Airflow > Issue Type: Sub-task > Components: models >Affects Versions: 1.10.1 >Reporter: Fokko Driesprong >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] OmerJog commented on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator
OmerJog commented on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator URL: https://github.com/apache/airflow/pull/6565#issuecomment-553302474 @yimingl17 you have an error `airflow/operators/sql_to_gcs.py:257:63: E0602: Undefined variable 'string_types' (undefined-variable)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5442) Druid broker hook get pandas DataFrame implementation
[ https://issues.apache.org/jira/browse/AIRFLOW-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5442: -- Fix Version/s: (was: 1.10.6) 1.10.7 > Druid broker hook get pandas DataFrame implementation > - > > Key: AIRFLOW-5442 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5442 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.5 >Reporter: Sayed Mohammad Hossein Torabi >Assignee: Sayed Mohammad Hossein Torabi >Priority: Minor > Fix For: 1.10.7 > > > The *get_pandas_df* of *DruidDbApiHook* returns NotImplementedError. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6566: [AIRFLOW-5917][part of AIRFLOW-5893] Group tests for the Webserver command
potiuk merged pull request #6566: [AIRFLOW-5917][part of AIRFLOW-5893] Group tests for the Webserver command URL: https://github.com/apache/airflow/pull/6566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6547: [AIRFLOW-5898] fix alembic crash due to typing import
potiuk commented on issue #6547: [AIRFLOW-5898] fix alembic crash due to typing import URL: https://github.com/apache/airflow/pull/6547#issuecomment-553316345 Thanks @houqp ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #6547: [AIRFLOW-5898] fix alembic crash due to typing import
potiuk merged pull request #6547: [AIRFLOW-5898] fix alembic crash due to typing import URL: https://github.com/apache/airflow/pull/6547 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5917) Group tests for the Webserver command
[ https://issues.apache.org/jira/browse/AIRFLOW-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973169#comment-16973169 ] ASF GitHub Bot commented on AIRFLOW-5917: - potiuk commented on pull request #6566: [AIRFLOW-5917][part of AIRFLOW-5893] Group tests for the Webserver command URL: https://github.com/apache/airflow/pull/6566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Group tests for the Webserver command > - > > Key: AIRFLOW-5917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5917 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5898) alembic command crashes due to typing import path collision
[
https://issues.apache.org/jira/browse/AIRFLOW-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973172#comment-16973172
]
ASF GitHub Bot commented on AIRFLOW-5898:
-
potiuk commented on pull request #6547: [AIRFLOW-5898] fix alembic crash due to
typing import
URL: https://github.com/apache/airflow/pull/6547
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> alembic command crashes due to typing import path collision
> ---
>
> Key: AIRFLOW-5898
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
> Project: Apache Airflow
> Issue Type: New Feature
> Components: database
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> Running alemibc within airflow directory results in the following crash:
>
> {code:java}
> File
> "/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py",
> line 17, in
> from typing import Generic, Callable, TypeVar, Tuple
> ImportError: cannot import name 'Generic' from 'typing'
> {code}
> This is due to typing.py within airlfow directory collides with the builtin
> typing module when a command runs inside airflow directory. Renaming it to
> `custom_typing.py` fixes the issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5917) Group tests for the Webserver command
[ https://issues.apache.org/jira/browse/AIRFLOW-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973170#comment-16973170 ] ASF subversion and git services commented on AIRFLOW-5917: -- Commit 8b537c57f9278b9330ab1a5cb4ccf0c35ee68ee9 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8b537c5 ] [AIRFLOW-5917][part of AIRFLOW-5893] Group tests for the Webserver command (#6566) > Group tests for the Webserver command > - > > Key: AIRFLOW-5917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5917 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5893) Refactor CLI tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973171#comment-16973171 ] ASF subversion and git services commented on AIRFLOW-5893: -- Commit 8b537c57f9278b9330ab1a5cb4ccf0c35ee68ee9 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8b537c5 ] [AIRFLOW-5917][part of AIRFLOW-5893] Group tests for the Webserver command (#6566) > Refactor CLI tests > -- > > Key: AIRFLOW-5893 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5893 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Hello > Currently, CLI tests are not ordered, which makes it difficult to find the > correct test, and also makes it difficult to introduce further improvements. > Currently CLI tests are in the following classes: > * tests.cli.test_cli.TestCLI > * tests.core.TestCli > I would like to take action to group the tests according to the base command, > which creates the following classes: > * tests.cli.test_cli.TestConnections > * tests.cli.test_cli.TestDags > * tests.cli.test_cli.TestDb > * tests.cli.test_cli.TestFlower > * tests.cli.test_cli.TestKerberos > * tests.cli.test_cli.TestPools > * tests.cli.test_cli.TestRoles > * tests.cli.test_cli.TestRotateFernetKey > * tests.cli.test_cli.TestScheduler > * tests.cli.test_cli.TestServeLogs > * tests.cli.test_cli.TestSyncPerm > * tests.cli.test_cli.TestTasks > * tests.cli.test_cli.TestUsers > * tests.cli.test_cli.TestVariables > * tests.cli.test_cli.TestVersion > * tests.cli.test_cli.TestWebserver > * tests.cli.test_cli.TestWorker > The change will make it easier to make subsequent changes for this component: > In particular, split the component cli.py file into separate modules, and > then introduce lazy module loading. Currently, all modules are loaded with > all dependencies every time. This process is time consuming. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5898) alembic command crashes due to typing import path collision
[
https://issues.apache.org/jira/browse/AIRFLOW-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973173#comment-16973173
]
ASF subversion and git services commented on AIRFLOW-5898:
--
Commit 3a3730eaa5ff1b6ca25a510bd7fb6894f8965bc3 in airflow's branch
refs/heads/master from Qingping Hou
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3a3730e ]
[AIRFLOW-5898] fix alembic crash due to typing import (#6547)
> alembic command crashes due to typing import path collision
> ---
>
> Key: AIRFLOW-5898
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
> Project: Apache Airflow
> Issue Type: New Feature
> Components: database
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> Running alemibc within airflow directory results in the following crash:
>
> {code:java}
> File
> "/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py",
> line 17, in
> from typing import Generic, Callable, TypeVar, Tuple
> ImportError: cannot import name 'Generic' from 'typing'
> {code}
> This is due to typing.py within airlfow directory collides with the builtin
> typing module when a command runs inside airflow directory. Renaming it to
> `custom_typing.py` fixes the issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] acroos commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request
acroos commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request URL: https://github.com/apache/airflow/pull/6380#issuecomment-553318316 @deshraj yeah sorry I've been sick for a little while now, I'll try to fix it ASAP when I can This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5917) Group tests for the Webserver command
[ https://issues.apache.org/jira/browse/AIRFLOW-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5917. --- Fix Version/s: 2.0.0 Resolution: Fixed > Group tests for the Webserver command > - > > Key: AIRFLOW-5917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5917 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5898) alembic command crashes due to typing import path collision
[
https://issues.apache.org/jira/browse/AIRFLOW-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-5898.
---
Fix Version/s: 2.0.0
Resolution: Fixed
> alembic command crashes due to typing import path collision
> ---
>
> Key: AIRFLOW-5898
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
> Project: Apache Airflow
> Issue Type: New Feature
> Components: database
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Running alemibc within airflow directory results in the following crash:
>
> {code:java}
> File
> "/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py",
> line 17, in
> from typing import Generic, Callable, TypeVar, Tuple
> ImportError: cannot import name 'Generic' from 'typing'
> {code}
> This is due to typing.py within airlfow directory collides with the builtin
> typing module when a command runs inside airflow directory. Renaming it to
> `custom_typing.py` fixes the issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region
[
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973189#comment-16973189
]
Paul De commented on AIRFLOW-5249:
--
The latest version avail on GCP (Cloud Composer service) is 1.10.3 and I get
the same error (404 Job not found) in europe-west2 for
BigQueryIntervalCheckOperator using 1.10.3
It's a significant limitation - if possible, a fix would be good.
> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on
> a dataset that is not in the 'US' location my task fails with the following
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status
> check failed. Final error was: 404
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
> returned "Not found: Job
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py",
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py",
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py",
> line 138, in get_first
> cur.execute(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1821, in execute
> self.job_id = self.run_query(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 849, in run_query
> return self.run_with_configuration(configuration)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without
> specifying a location. When I run the same operator on a dataset that is in
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments
> and does not pass a location to the BigQueryHook, I believe this is the
> source of the problem.
>
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to
> this one, but the referenced task calls out the two operators I'm having an
> issue with as out of scope and after commenting on that task I have not
> received a response.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5873) KubernetesPodOperator fixes and test
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David de L. updated AIRFLOW-5873: - Description: There are two PR's for this task, one targeted at v1-10-test branch, and one adapted to master branch. * `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) * `resources` passed to PodGenerator [should be `k8s.V1ResourceRequirements`]([https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45]), which is now handled in `KubernetesPodOperator` * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts arbitrary values in `setattr` (not present on either branch https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/contrib/operators/kubernetes_pod_operator.py#L166-L171) * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and refactors ** e.g. move kubernetes folder one level up from `/contrib` [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21] ** fix `xcom_push` to `do_xcom_push` [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90] was: * `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) * `resources` passed to PodGenerator [should be `k8s.V1ResourceRequirements`](https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45), which is now handled in `KubernetesPodOperator` * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts arbitrary values in `setattr` (present on `v1-10-test` branch [https://github.com/apache/airflow/blame/v1-10-test/airflow/contrib/operators/kubernetes_pod_operator.py#L166]) * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and refactors ** e.g. move kubernetes folder one level up from `/contrib` [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21] ** fix `xcom_push` to `do_xcom_push` [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90] > KubernetesPodOperator fixes and test > > > Key: AIRFLOW-5873 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5873 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.6 >Reporter: David de L. >Assignee: David de L. >Priority: Major > Fix For: 2.0.0, 1.10.7 > > > There are two PR's for this task, one targeted at v1-10-test branch, and one > adapted to master branch. > * `KubernetesPodOperator` kwarg `resources` is erroneously passed to > `base_operator`, instead should only go to `PodGenerator`. The two have > different syntax. (both on `master` and `v1-10-test` branches) > * `resources` passed to PodGenerator [should be > `k8s.V1ResourceRequirements`]([https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45]), > which is now handled in `KubernetesPodOperator` > * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts > arbitrary values in `setattr` (not present on either branch > https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/contrib/operators/kubernetes_pod_operator.py#L166-L171) > * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and > refactors > ** e.g. move kubernetes folder one level up from `/contrib` > [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21] > ** fix `xcom_push` to `do_xcom_push` > [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) S3 to Mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Affects Version/s: 2.0.0 > S3 to Mysql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5908) Add download_file to S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5908: -- Affects Version/s: 2.0.0 > Add download_file to S3 Hook > > > Key: AIRFLOW-5908 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region
[
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973189#comment-16973189
]
Paul De edited comment on AIRFLOW-5249 at 11/13/19 9:59 AM:
The latest version avail on GCP (Cloud Composer service) is 1.10.3 and I get
the same error (404 Job not found) in europe-west2 for
BigQueryIntervalCheckOperator using 1.10.3
It's a significant limitation and GCP does patch recent versions
[https://cloud.google.com/composer/docs/release-notes]
- if possible, a fix would be good.
was (Author: pauld):
The latest version avail on GCP (Cloud Composer service) is 1.10.3 and I get
the same error (404 Job not found) in europe-west2 for
BigQueryIntervalCheckOperator using 1.10.3
It's a significant limitation - if possible, a fix would be good.
> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on
> a dataset that is not in the 'US' location my task fails with the following
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status
> check failed. Final error was: 404
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
> returned "Not found: Job
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py",
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py",
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py",
> line 138, in get_first
> cur.execute(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1821, in execute
> self.job_id = self.run_query(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 849, in run_query
> return self.run_with_configuration(configuration)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without
> specifying a location. When I run the same operator on a dataset that is in
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments
> and does not pass a location to the BigQueryHook, I believe this is the
> source of the problem.
>
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to
> this one, but the referenced task calls out the two operators I'm having an
> issue with as out of scope and after commenting on that task I have not
> received a response.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
Felix Uellendall created AIRFLOW-5921: - Summary: Add bulk_load_custom to MySqlHook Key: AIRFLOW-5921 URL: https://issues.apache.org/jira/browse/AIRFLOW-5921 Project: Apache Airflow Issue Type: New Feature Components: hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6549: [AIRFLOW-5880] Enforce unique task ids
codecov-io commented on issue #6549: [AIRFLOW-5880] Enforce unique task ids URL: https://github.com/apache/airflow/pull/6549#issuecomment-553334514 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=h1) Report > Merging [#6549](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr&el=desc) will **decrease** coverage by `0.76%`. > The diff coverage is `85.71%`. [](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6549 +/- ## == - Coverage 84.07% 83.31% -0.77% == Files 639 646 +7 Lines 3690037298 +398 == + Hits3102431073 +49 - Misses 5876 6225 +349 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.21% <100%> (+0.27%)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.85% <66.66%> (-0.02%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.37%)` | :arrow_down: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/operators/bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmFzaF9vcGVyYXRvci5weQ==) | `90.56% <0%> (-0.82%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/airflow/pull/6549/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=footer). Last update [d25ef2b...106dc72](https://codecov.io/gh/apache/airflow/pull/6549?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6558: [AIRFLOW-5905] Unify GCP file names
codecov-io commented on issue #6558: [AIRFLOW-5905] Unify GCP file names URL: https://github.com/apache/airflow/pull/6558#issuecomment-553336156 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=h1) Report > Merging [#6558](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e503544b0ca45654b40f2f7b30ae22f2b62b89fa?src=pr&el=desc) will **decrease** coverage by `0.25%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6558 +/- ## == - Coverage 83.57% 83.32% -0.26% == Files 643 646 +3 Lines 3726337360 +97 == - Hits3114431129 -15 - Misses 6119 6231 +112 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `72.89% <0%> (-25.62%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [airflow/operators/bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmFzaF9vcGVyYXRvci5weQ==) | `90.56% <0%> (-0.82%)` | :arrow_down: | | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `91.46% <0%> (-0.14%)` | :arrow_down: | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `94.63% <0%> (-0.07%)` | :arrow_down: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <0%> (ø)` | :arrow_up: | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/6558/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=footer). Last update [e503544...77916ec](https://codecov.io/gh/apache/airflow/pull/6558?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kgabryje opened a new pull request #156: enabled adding integration logo to integration box
kgabryje opened a new pull request #156: enabled adding integration logo to integration box URL: https://github.com/apache/airflow-site/pull/156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6558: [AIRFLOW-5905] Unify GCP file names
mik-laj merged pull request #6558: [AIRFLOW-5905] Unify GCP file names URL: https://github.com/apache/airflow/pull/6558 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5905) Unify GCP file names
[ https://issues.apache.org/jira/browse/AIRFLOW-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973227#comment-16973227 ] ASF GitHub Bot commented on AIRFLOW-5905: - mik-laj commented on pull request #6558: [AIRFLOW-5905] Unify GCP file names URL: https://github.com/apache/airflow/pull/6558 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unify GCP file names > > > Key: AIRFLOW-5905 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5905 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5905) Unify GCP file names
[ https://issues.apache.org/jira/browse/AIRFLOW-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5905. Fix Version/s: 2.0.0 Resolution: Fixed > Unify GCP file names > > > Key: AIRFLOW-5905 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5905 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Trivial > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5905) Unify GCP file names
[ https://issues.apache.org/jira/browse/AIRFLOW-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973229#comment-16973229 ] ASF subversion and git services commented on AIRFLOW-5905: -- Commit 8c3d8936dcc85d80a75f70be2293980d7e894cb5 in airflow's branch refs/heads/master from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8c3d893 ] [AIRFLOW-5905] Unify GCP file names (#6558) > Unify GCP file names > > > Key: AIRFLOW-5905 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5905 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r345693428 ## File path: airflow/utils/state.py ## @@ -99,6 +99,7 @@ def finished(cls): cls.SUCCESS, cls.FAILED, cls.SKIPPED, +cls.UPSTREAM_FAILED Review comment: Great, I will change it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] GnunuX commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
GnunuX commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553351901 Checks ave failed for an unknown error. I think this pull request is ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy
potiuk opened a new pull request #6570: [AIRFLOW-5886] - fixed a bit too
selective Dockerfile copy
URL: https://github.com/apache/airflow/pull/6570
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5886
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[
https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973248#comment-16973248
]
ASF GitHub Bot commented on AIRFLOW-5886:
-
potiuk commented on pull request #6570: [AIRFLOW-5886] - fixed a bit too
selective Dockerfile copy
URL: https://github.com/apache/airflow/pull/6570
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5886
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Copying sources in Dockerfile can be selective
> --
>
> Key: AIRFLOW-5886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5886
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.7
>
>
> This change will further improve time of rebuilds for docker image when your
> sources change (very useful in case of building kubernetes image). It adds
> only directories that are needed (it is synchronised with .dockerignore and
> local mounts) and in the sequence that reflects frequency of changes. Also
> pip install is not done again after sources change (there is no point) so the
> build is much faster when only sources or test file change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-5886: --- > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553355205 We have a master failing. Fix is on the way. I will ask you to rebase after this is merged. https://github.com/apache/airflow/pull/6570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553355766 sorry for the trouble - it was my mistake / merges and rebases of many issues yesterday This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973250#comment-16973250 ] ASF subversion and git services commented on AIRFLOW-5886: -- Commit c65716041bd2bc4d74265a1e4110ed352f370f05 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c657160 ] [AIRFLOW-5886] Selective copying of sources in Docker image (#6538) This change further improves time of rebuilds for docker image when your sources change (very useful in case of building kubernetes image). It adds only directories that are needed (it is synchronised with .dockerignore and local mounts) and in the sequence that reflects frequency of changes. Also pip install is not done again after sources change (there is no point) so the build is much faster when only sources or test file change. (cherry picked from commit cc754f7f74a741e01277bea6d5586e1dd6d64af8) > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy
potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy URL: https://github.com/apache/airflow/pull/6570#issuecomment-553358760 It was a bit too selective change - I have to copy the whole CI folder as we also need the CI scripts for DinD Kind operations - i.e. docker builds executed inside the docker container :). I think I must have seen a wrong CI output and merged it hastily yesterday (among 15+ merges I've done yesterday :). I'd fix it without waiting for Travis so that some PRs can be rebased quickly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] GnunuX commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
GnunuX commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553360358 No problem, thanks for your reply. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6563: [AIRFLOW-5915] Add support for the new documentation theme
codecov-io commented on issue #6563: [AIRFLOW-5915] Add support for the new documentation theme URL: https://github.com/apache/airflow/pull/6563#issuecomment-553361055 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=h1) Report > Merging [#6563](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/f2473b320f73d19e60a6f9502bc92538f6587dce?src=pr&el=desc) will **decrease** coverage by `0.23%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6563 +/- ## == - Coverage 83.53% 83.29% -0.24% == Files 645 645 Lines 3728637291 +5 == - Hits3114731063 -84 - Misses 6139 6228 +89 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.37%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `57.99% <0%> (-0.33%)` | :arrow_down: | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `94.69% <0%> (ø)` | :arrow_up: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/6563/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=footer). Last update [f2473b3...2b1de0b](https://codecov.io/gh/apache/airflow/pull/6563?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request
codecov-io edited a comment on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request URL: https://github.com/apache/airflow/pull/6380#issuecomment-544885909 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=h1) Report > Merging [#6380](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4a90ba05bc55fab6634a6a7f2f2957857b1f18e3?src=pr&el=desc) will **decrease** coverage by `74.32%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master #6380 +/- ## == - Coverage83.9% 9.57% -74.33% == Files 635 635 Lines 36678 36684+6 == - Hits307733512-27261 - Misses 5905 33172+27267 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/strings.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdHJpbmdzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `0% <0%> (-88.24%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/aws/hooks/sqs.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXdzL2hvb2tzL3Nxcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/sensors/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avc2Vuc29ycy9iaWdxdWVyeV9kdHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/operators/text\_to\_speech.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL3RleHRfdG9fc3BlZWNoLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [569 more](https://codecov.io/gh/apache/airflow/pull/6380/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=footer). Last update [4a90ba0...3efbe9c](https://codecov.io/gh/apache/airflow/pull/6380?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553363016 Fun fact for the curious ones - it was not detected yesterday because of a the weird Kubernetes test dependency we have. Currently Kubernetes tests are testing the PREVIOUS MASTER Docker image not the CURRENT SOURCES (not good) so when the PR was run, it was Green as previous Docker file had all the files and the new one did not. So after merge it kind of tripped over its own feet and got broken. Luckily the permanent fix for the Kubernetes “previous master vs. current” is coming in https://github.com/apache/airflow/pull/6516 which is next in line to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy
potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy URL: https://github.com/apache/airflow/pull/6570#issuecomment-553363362 Fun fact for the curious ones - I know the reason it was Green yesterday - because of a the weird Kubernetes test dependency we have. Currently Kubernetes tests are testing the PREVIOUS MASTER Docker image not the CURRENT SOURCES (not good) so when the PR was run, it was Green as previous Docker file had all the files and the new one did not. So after merge it kind of tripped over its own feet and got broken. Luckily the permanent fix for the Kubernetes “previous master vs. current” is coming in https://github.com/apache/airflow/pull/6516 which is next in line to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kgabryje opened a new pull request #157: enabled hiding sidebar by setting a variable in frontmatter
kgabryje opened a new pull request #157: enabled hiding sidebar by setting a variable in frontmatter URL: https://github.com/apache/airflow-site/pull/157 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6569: [AIRFLOW-5916] ADDED PASSWORD-ALIAS EXTRA OPTION
codecov-io commented on issue #6569: [AIRFLOW-5916] ADDED PASSWORD-ALIAS EXTRA OPTION URL: https://github.com/apache/airflow/pull/6569#issuecomment-553368588 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=h1) Report > Merging [#6569](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c481d705a599bd12baf33b966a80858182021128?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `66.66%`. [](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6569 +/- ## == - Coverage 83.31% 83.31% -0.01% == Files 645 647 +2 Lines 3729137316 +25 == + Hits3106831088 +20 - Misses 6223 6228 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/sqoop\_hook.py](https://codecov.io/gh/apache/airflow/pull/6569/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nxb29wX2hvb2sucHk=) | `94.93% <66.66%> (-0.55%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6569/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6569/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `57.99% <0%> (-0.17%)` | :arrow_down: | | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6569/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | | | [airflow/typing\_compat.py](https://codecov.io/gh/apache/airflow/pull/6569/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmdfY29tcGF0LnB5) | `100% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=footer). Last update [c481d70...ab5201b](https://codecov.io/gh/apache/airflow/pull/6569?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy
mik-laj merged pull request #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy URL: https://github.com/apache/airflow/pull/6570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973286#comment-16973286 ] ASF GitHub Bot commented on AIRFLOW-5886: - mik-laj commented on pull request #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy URL: https://github.com/apache/airflow/pull/6570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973287#comment-16973287 ] ASF subversion and git services commented on AIRFLOW-5886: -- Commit 2897624a510361948bb9e261dd2da7ab74271312 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2897624 ] [AIRFLOW-5886] Fixed a bit too selective Dockerfile copy (#6570) > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy
potiuk commented on issue #6570: [AIRFLOW-5886] - fixed a bit too selective Dockerfile copy URL: https://github.com/apache/airflow/pull/6570#issuecomment-553384357 Thanks @mik-laj ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
codecov-io edited a comment on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-54999 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=h1) Report > Merging [#6371](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3a3730eaa5ff1b6ca25a510bd7fb6894f8965bc3?src=pr&el=desc) will **increase** coverage by `0.19%`. > The diff coverage is `86.91%`. [](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6371 +/- ## = + Coverage83.3% 83.5% +0.19% = Files 645 645 Lines 37291 37356 +65 = + Hits31067 31194 +127 + Misses 62246162 -62 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...ders/google/cloud/example\_dags/example\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFwcm9jLnB5) | `0% <0%> (ø)` | | | [airflow/contrib/hooks/gcp\_dataproc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhcHJvY19ob29rLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/operators/dataproc\_operator.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhcHJvY19vcGVyYXRvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...rflow/providers/google/cloud/operators/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kYXRhcHJvYy5weQ==) | `88.27% <93.91%> (ø)` | | | [airflow/providers/google/cloud/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2RhdGFwcm9jLnB5) | `98.6% <98.6%> (ø)` | | | [...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==) | | | | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6371/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=footer). Last update [3a3730e...ec33459](https://codecov.io/gh/apache/airflow/pull/6371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 closed pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 closed pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973340#comment-16973340 ] ASF GitHub Bot commented on AIRFLOW-3607: - amichai07 commented on pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Decreasing scheduler delay between tasks > > > Key: AIRFLOW-3607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3607 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.0, 1.10.1, 1.10.2 > Environment: ubuntu 14.04 >Reporter: Amichai Horvitz >Assignee: Amichai Horvitz >Priority: Major > Original Estimate: 336h > Remaining Estimate: 336h > > I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) > that says instead of checking the query for every task let the tasks report > to the dagrun. I have a dag with many tasks and the delay between tasks can > rise to 10 seconds or more, I already changed the configuration, added > processes and memory, checked the code and did research, profiling and other > experiments. I hope that this change will make a drastic change in the delay. > I would be happy to discuss this solution, the research and other solutions > for this issue. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] amichai07 opened a new pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 opened a new pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751 ### Jira * My PR addresses the following https://issues.apache.org/jira/browse/AIRFLOW-3607 and references them in the PR title * Decreasing scheduler delay between tasks ### Description * The delay between tasks can be a major issue, especially when we have dags with many subdags, figures out that the scheduling process spends plenty of time in dependency checking, we took the trigger rule dependency which calls the db for each task instance, we made it call the db just once for each dag_run. ### Tests * My pr uses very basic parts of the code and also has a fall back to the original behaviour, the ci covers most of the logic and cases that might happen already, I added a unit/integration test that checks the function that has most of the new logic ### Commits * removed unnecessary queries - run on each dag run instead of each ti ### Documentation no need for new docs ### Code Quality * Passes `flake8` * Tested in production environment for 3 days ### Results The tests was made on a heavily multitasks dag (35 tasks). The tasks don't do any db queries **On local environment** before changes: - avg delay between tasks: 4.22 sec - number of queries during 10 minutes: 118,879 after collecting dep check queries: - avg delay between tasks: 3.86 sec - number of queries during 10 minutes: 104,397 Stress test - running the dag for every 10 sec for an hour: before changes: - avg delay between tasks: 16.7 sec - number of queries: 943,230 after: - avg delay between tasks: 3.28 sec - number of queries: 734,563 **On production environment** before changes: - avg delay between tasks: 2.45 sec after: - avg delay between tasks: 2.16 sec Stress test - running with 150 other dags: before changes: - avg delay between tasks: 6.3 sec after: - avg delay between tasks: 4.74 sec ### Edit we recently did a stress test to check this change again with version 1.10.4 we did the test on a staging like production environment with one dag with 49 tasks that starts once a minute : - max delay was decreased by 30% - db cpu was decreased by 20% - avg delay was decreased 18% This is a graph of the time delay between two tasks in the dag by time: - With improvement(first segment): 14:30-16:30 - Without improvement(last segment): 18:10-20:10 https://user-images.githubusercontent.com/6268317/68576601-10575680-0477-11ea-8928-35c210cc903e.png";> This graph shows the db read iops difference which can explain the change (the times are different because this is in utc)  ### Conclusion The query that we changed indeed had a dramatic impact on the performance of the scheduler. Reusing db results decreased the delay notably and gave the system chance to recover from stress. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973350#comment-16973350 ] ASF GitHub Bot commented on AIRFLOW-3607: - amichai07 commented on pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751 ### Jira * My PR addresses the following https://issues.apache.org/jira/browse/AIRFLOW-3607 and references them in the PR title * Decreasing scheduler delay between tasks ### Description * The delay between tasks can be a major issue, especially when we have dags with many subdags, figures out that the scheduling process spends plenty of time in dependency checking, we took the trigger rule dependency which calls the db for each task instance, we made it call the db just once for each dag_run. ### Tests * My pr uses very basic parts of the code and also has a fall back to the original behaviour, the ci covers most of the logic and cases that might happen already, I added a unit/integration test that checks the function that has most of the new logic ### Commits * removed unnecessary queries - run on each dag run instead of each ti ### Documentation no need for new docs ### Code Quality * Passes `flake8` * Tested in production environment for 3 days ### Results The tests was made on a heavily multitasks dag (35 tasks). The tasks don't do any db queries **On local environment** before changes: - avg delay between tasks: 4.22 sec - number of queries during 10 minutes: 118,879 after collecting dep check queries: - avg delay between tasks: 3.86 sec - number of queries during 10 minutes: 104,397 Stress test - running the dag for every 10 sec for an hour: before changes: - avg delay between tasks: 16.7 sec - number of queries: 943,230 after: - avg delay between tasks: 3.28 sec - number of queries: 734,563 **On production environment** before changes: - avg delay between tasks: 2.45 sec after: - avg delay between tasks: 2.16 sec Stress test - running with 150 other dags: before changes: - avg delay between tasks: 6.3 sec after: - avg delay between tasks: 4.74 sec ### Edit we recently did a stress test to check this change again with version 1.10.4 we did the test on a staging like production environment with one dag with 49 tasks that starts once a minute : - max delay was decreased by 30% - db cpu was decreased by 20% - avg delay was decreased 18% This is a graph of the time delay between two tasks in the dag by time: - With improvement(first segment): 14:30-16:30 - Without improvement(last segment): 18:10-20:10 https://user-images.githubusercontent.com/6268317/68576601-10575680-0477-11ea-8928-35c210cc903e.png";> This graph shows the db read iops difference which can explain the change (the times are different because this is in utc)  ### Conclusion The query that we changed indeed had a dramatic impact on the performance of the scheduler. Reusing db results decreased the delay notably and gave the system chance to recover from stress. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Decreasing scheduler delay between tasks > > > Key: AIRFLOW-3607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3607 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.0, 1.10.1, 1.10.2 > Environment: ubuntu 14.04 >Reporter: Amichai Horvitz >Assignee: Amichai Horvitz >Priority: Major > Original Estimate: 336h > Remaining Estimate: 336h > > I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) > that says instead of checking the query for every task let the tasks report > to the dagrun. I have a dag with many tasks and the delay between tasks can > rise to 10 seconds or more, I already changed the configuration, added > processes and memory, checked the code and did research, profiling and other > experiments. I hope that this change will make a drastic change in the delay. > I would be happy to discuss this solution, the research and other solutions > for this issue. > Thanks -- This message was sent by Atla
[GitHub] [airflow] potiuk commented on issue #6562: Avoid warning from abc.collections
potiuk commented on issue #6562: Avoid warning from abc.collections URL: https://github.com/apache/airflow/pull/6562#issuecomment-553411949 I think it's better to have the PR to apache:v1-10-test @ingenieroariel -> we usually put everything in the v1-10-test (we cherry-pick changes there directly so sometimes it gets broken) - and only when we are readying for 1.10.* releases we move stable branch to the tip of v1-10-test. That would be easiest thing to do for us otherwise we have to compare stable/test branches. :). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6562: Avoid warning from abc.collections
potiuk commented on issue #6562: Avoid warning from abc.collections URL: https://github.com/apache/airflow/pull/6562#issuecomment-553412197 Also v1-10-test branch is now quite ahead of v1-10-stable so you might want to use it as there are already some changes there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
potiuk commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553412932 OK. You can rebase on the current master @GnunuX . The master build has not yet finished but it's past the point the problem occured: https://travis-ci.org/apache/airflow/jobs/611340042?utm_medium=notification&utm_source=github_status This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
codecov-io commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-553413704 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=h1) Report > Merging [#5785](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c481d705a599bd12baf33b966a80858182021128?src=pr&el=desc) will **increase** coverage by `0.02%`. > The diff coverage is `96.77%`. [](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#5785 +/- ## == + Coverage 83.31% 83.34% +0.02% == Files 645 647 +2 Lines 3729137353 +62 == + Hits3106831130 +62 Misses 6223 6223 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/azure\_data\_explorer.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9kYXRhX2V4cGxvcmVyLnB5) | `100% <100%> (ø)` | | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `90.26% <100%> (+0.08%)` | :arrow_up: | | [airflow/contrib/hooks/azure\_data\_explorer\_hook.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfZXhwbG9yZXJfaG9vay5weQ==) | `94.87% <94.87%> (ø)` | | | [...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==) | | | | [airflow/typing.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmcucHk=) | | | | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | | | [airflow/typing\_compat.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmdfY29tcGF0LnB5) | `100% <0%> (ø)` | | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=footer). Last update [c481d70...d37a08c](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
codecov-io edited a comment on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-553413704 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=h1) Report > Merging [#5785](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c481d705a599bd12baf33b966a80858182021128?src=pr&el=desc) will **increase** coverage by `0.02%`. > The diff coverage is `96.77%`. [](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#5785 +/- ## == + Coverage 83.31% 83.34% +0.02% == Files 645 647 +2 Lines 3729137353 +62 == + Hits3106831130 +62 Misses 6223 6223 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <ø> (ø)` | :arrow_up: | | [airflow/contrib/operators/azure\_data\_explorer.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9kYXRhX2V4cGxvcmVyLnB5) | `100% <100%> (ø)` | | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `90.26% <100%> (+0.08%)` | :arrow_up: | | [airflow/contrib/hooks/azure\_data\_explorer\_hook.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfZXhwbG9yZXJfaG9vay5weQ==) | `94.87% <94.87%> (ø)` | | | [...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==) | | | | [airflow/typing.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmcucHk=) | | | | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | | | [airflow/typing\_compat.py](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmdfY29tcGF0LnB5) | `100% <0%> (ø)` | | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/5785/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=footer). Last update [c481d70...d37a08c](https://codecov.io/gh/apache/airflow/pull/5785?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5886. --- Resolution: Fixed > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
Felix Uellendall created AIRFLOW-5922: - Summary: Add option to specify the mysql client library used in MySqlHook Key: AIRFLOW-5922 URL: https://issues.apache.org/jira/browse/AIRFLOW-5922 Project: Apache Airflow Issue Type: New Feature Components: dependencies, hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator
hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture
task logs in DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#discussion_r345737656
##
File path: airflow/contrib/operators/docker_swarm_operator.py
##
@@ -127,7 +133,37 @@ def _run_image(self):
# wait for the service to start the task
while not self.cli.tasks(filters={'service': self.service['ID']}):
continue
+
+logs = self.cli.service_logs(self.service['ID'], follow=True,
stdout=True, stderr=True, is_tty=self.tty)
+line = ''
+_stream_logs = self.enable_logging # Status of the service_logs'
generator
while True:
+if self.enable_logging:
+while _stream_logs:
+try:
+log = next(logs)
+# TODO: Remove this clause once
https://github.com/docker/docker-py/issues/931 is fixed
+except (requests.exceptions.ConnectionError) as e:
Review comment:
`except requests.exceptions.ConnectionError`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator
hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator URL: https://github.com/apache/airflow/pull/6552#discussion_r345775385 ## File path: airflow/contrib/operators/docker_swarm_operator.py ## @@ -89,17 +90,22 @@ class DockerSwarmOperator(DockerOperator): :param tty: Allocate pseudo-TTY to the container of this service This needs to be set see logs of the Docker container / service. :type tty: bool +:param enable_logging: Show the application's logs in operator's logs. +Supported only if the Docker engine is using json-file or journald logging drivers. +The `tty` parameter should be set to use this with Python applications. +:type enable_logging: bool """ @apply_defaults def __init__( self, image, +enable_logging=True, *args, **kwargs): - super().__init__(image=image, *args, **kwargs) +self.enable_logging = enable_logging self.service = None def _run_image(self): Review comment: Not really related to the PR but I found this `types.Resources(mem_limit=self.mem_limit)` should better be `types.Resources(mem_limit=docker.utils.parse_bytes(self.mem_limit))` or we get non-parseable errors from the API (becomes float which is bad or string which is also bad) Appears to work in your test though... :confused: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator
hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture
task logs in DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#discussion_r345737517
##
File path: airflow/contrib/operators/docker_swarm_operator.py
##
@@ -127,7 +133,37 @@ def _run_image(self):
# wait for the service to start the task
while not self.cli.tasks(filters={'service': self.service['ID']}):
continue
+
+logs = self.cli.service_logs(self.service['ID'], follow=True,
stdout=True, stderr=True, is_tty=self.tty)
+line = ''
+_stream_logs = self.enable_logging # Status of the service_logs'
generator
while True:
+if self.enable_logging:
+while _stream_logs:
+try:
+log = next(logs)
+# TODO: Remove this clause once
https://github.com/docker/docker-py/issues/931 is fixed
+except (requests.exceptions.ConnectionError) as e:
+# If the service log stream stopped sending messages,
check if it the service has terminated.
+break
+except (StopIteration) as e:
Review comment:
`except StopIteration`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator
hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator URL: https://github.com/apache/airflow/pull/6552#discussion_r345736792 ## File path: airflow/contrib/operators/docker_swarm_operator.py ## @@ -127,7 +133,37 @@ def _run_image(self): # wait for the service to start the task Review comment: `status = None` can be removed now I guess This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator
hredestig commented on a change in pull request #6552: AIRFLOW-5850: Capture
task logs in DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#discussion_r345776382
##
File path: airflow/contrib/operators/docker_swarm_operator.py
##
@@ -127,7 +133,37 @@ def _run_image(self):
# wait for the service to start the task
while not self.cli.tasks(filters={'service': self.service['ID']}):
continue
+
+logs = self.cli.service_logs(self.service['ID'], follow=True,
stdout=True, stderr=True, is_tty=self.tty)
+line = ''
+_stream_logs = self.enable_logging # Status of the service_logs'
generator
while True:
Review comment:
Maybe I do something wrong but with
```
DockerSwarmOperator(
task_id='blub',
default_args=default_args,
environment=dict(os.environ),
docker_url=os.environ['DOCKER_MASTER_URL'],
command="""python -c 'import sys; print("should kick of workflow
here"); sys.exit(0)'""",
image='python:3.7',
auto_remove=True,
dag=dag,
)
```
My service just hangs indefinitely...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #6097: [AIRFLOW-5478] Decode PythonVirtualenvOperator Output to Logs
OmerJog commented on issue #6097: [AIRFLOW-5478] Decode PythonVirtualenvOperator Output to Logs URL: https://github.com/apache/airflow/pull/6097#issuecomment-553421214 @ramannanda are you still working on this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
ashb commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-553422473 @yuqian90 Yes, it probably is safe to do that. Certainly if you try it and report back it will increas the chance that we would backport this to the next 1.10.x release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5112: [AIRFLOW-4209][AIP3-STEP13] Replace imp by importlib
ashb commented on issue #5112: [AIRFLOW-4209][AIP3-STEP13] Replace imp by importlib URL: https://github.com/apache/airflow/pull/5112#issuecomment-553423137 @brenoarosa The old module still exists and works, so this PR/the warning about deprecated imp module shouldn't be the source of any problems This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #6377: [AIRFLOW-5589] monitor pods by labels instead of names
kaxil edited a comment on issue #6377: [AIRFLOW-5589] monitor pods by labels instead of names URL: https://github.com/apache/airflow/pull/6377#issuecomment-553424695 @danccooper - FYI @dimberman is on holiday for a week, so I guess this will have to wait till he is back and has time to look at it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6377: [AIRFLOW-5589] monitor pods by labels instead of names
kaxil commented on issue #6377: [AIRFLOW-5589] monitor pods by labels instead of names URL: https://github.com/apache/airflow/pull/6377#issuecomment-553424695 @danccooper - FYI @dimberman is on holiday for a week This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6097: [AIRFLOW-5478] Decode PythonVirtualenvOperator Output to Logs
mik-laj commented on issue #6097: [AIRFLOW-5478] Decode PythonVirtualenvOperator Output to Logs URL: https://github.com/apache/airflow/pull/6097#issuecomment-553432923 Is it done on master? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
potiuk merged pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5691) Rewrite Dataproc operators to use python library
[ https://issues.apache.org/jira/browse/AIRFLOW-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973415#comment-16973415 ] ASF GitHub Bot commented on AIRFLOW-5691: - potiuk commented on pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rewrite Dataproc operators to use python library > > > Key: AIRFLOW-5691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5691 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5691) Rewrite Dataproc operators to use python library
[ https://issues.apache.org/jira/browse/AIRFLOW-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5691. --- Fix Version/s: 2.0.0 Resolution: Fixed > Rewrite Dataproc operators to use python library > > > Key: AIRFLOW-5691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5691 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5691) Rewrite Dataproc operators to use python library
[ https://issues.apache.org/jira/browse/AIRFLOW-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973416#comment-16973416 ] ASF subversion and git services commented on AIRFLOW-5691: -- Commit d633d3ac44c395e6c43cd388f98fba1ce1c435a3 in airflow's branch refs/heads/master from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d633d3a ] [AIRFLOW-5691] Rewrite Dataproc operators to use python library (#6371) > Rewrite Dataproc operators to use python library > > > Key: AIRFLOW-5691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5691 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow-site] kgabryje opened a new pull request #158: sorting versions in descending order
kgabryje opened a new pull request #158: sorting versions in descending order URL: https://github.com/apache/airflow-site/pull/158 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
codecov-io commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553451941 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=h1) Report > Merging [#6556](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d25ef2b2abc3da9f5725a3c433bc035b970eec23?src=pr&el=desc) will **decrease** coverage by `0.44%`. > The diff coverage is `44.59%`. [](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6556 +/- ## == - Coverage 84.07% 83.63% -0.45% == Files 639 645 +6 Lines 3690037291 +391 == + Hits3102431187 +163 - Misses 5876 6104 +228 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/typing\_compat.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy90eXBpbmdfY29tcGF0LnB5) | `100% <ø> (ø)` | | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `94.69% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGxwX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.47% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_datasync\_hook.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19kYXRhc3luY19ob29rLnB5) | `100% <100%> (ø)` | | | [airflow/contrib/operators/awsbatch\_operator.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hd3NiYXRjaF9vcGVyYXRvci5weQ==) | `78.4% <100%> (ø)` | :arrow_up: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `77.97% <100%> (ø)` | :arrow_up: | | [airflow/gcp/hooks/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnlfZHRzLnB5) | `86.2% <100%> (+0.24%)` | :arrow_up: | | [airflow/ti\_deps/deps/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | | | [airflow/contrib/hooks/gdrive\_hook.py](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dkcml2ZV9ob29rLnB5) | `100% <100%> (ø)` | :arrow_up: | | ... and [29 more](https://codecov.io/gh/apache/airflow/pull/6556/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=footer). Last update [d25ef2b...cd0d9f6](https://codecov.io/gh/apache/airflow/pull/6556?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
OmerJog commented on issue #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556#issuecomment-553452886 @GnunuX there is an issue with the rebase (Files changed 64) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kgabryje opened a new pull request #159: displaying anchor on hover on doc header instead of empty square
kgabryje opened a new pull request #159: displaying anchor on hover on doc header instead of empty square URL: https://github.com/apache/airflow-site/pull/159 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5923) Use absolute paths in GCP system tests
Tomasz Urbaszek created AIRFLOW-5923: Summary: Use absolute paths in GCP system tests Key: AIRFLOW-5923 URL: https://issues.apache.org/jira/browse/AIRFLOW-5923 Project: Apache Airflow Issue Type: Improvement Components: gcp, tests Affects Versions: 1.10.7 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6396: [AIRFLOW-5726] Delete table as file name in RedshiftToS3Transfer
codecov-io edited a comment on issue #6396: [AIRFLOW-5726] Delete table as file name in RedshiftToS3Transfer URL: https://github.com/apache/airflow/pull/6396#issuecomment-546875908 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=h1) Report > Merging [#6396](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/74d2a0d9e77cf90b85654f65d1adba0875e0fb1f?src=pr&el=desc) will **increase** coverage by `2.77%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6396 +/- ## == + Coverage 80.59% 83.37% +2.77% == Files 626 645 +19 Lines 3624337358+1115 == + Hits2921131146+1935 + Misses 7032 6212 -820 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.29% <100%> (+0.15%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=) | `63.33% <0%> (-31.12%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.37%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `81.81% <0%> (-11.37%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `86.88% <0%> (-5.54%)` | :arrow_down: | | ... and [196 more](https://codecov.io/gh/apache/airflow/pull/6396/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=footer). Last update [74d2a0d...bf6e4ee](https://codecov.io/gh/apache/airflow/pull/6396?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin opened a new pull request #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests
nuclearpinguin opened a new pull request #6571: [AIRFLOW-5923] Use absolute
paths in GCP system tests
URL: https://github.com/apache/airflow/pull/6571
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5923
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
In this PR I propose to use absolute path to example DAGs used in GCP system
tests. Used values should work well in breeze. I've also fixed system tests
that were moved to `providers.google.cloud`.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5923) Use absolute paths in GCP system tests
[
https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973481#comment-16973481
]
ASF GitHub Bot commented on AIRFLOW-5923:
-
nuclearpinguin commented on pull request #6571: [AIRFLOW-5923] Use absolute
paths in GCP system tests
URL: https://github.com/apache/airflow/pull/6571
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5923
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
In this PR I propose to use absolute path to example DAGs used in GCP system
tests. Used values should work well in breeze. I've also fixed system tests
that were moved to `providers.google.cloud`.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use absolute paths in GCP system tests
> --
>
> Key: AIRFLOW-5923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5923
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp, tests
>Affects Versions: 1.10.7
>Reporter: Tomasz Urbaszek
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
Felix Uellendall created AIRFLOW-5924: - Summary: Automatically unify bucket name and key in S3Hook Key: AIRFLOW-5924 URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 Project: Apache Airflow Issue Type: New Feature Components: aws, hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin opened a new pull request #6572: [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator
nuclearpinguin opened a new pull request #6572: [AIRFLOW-3489] Improve json
data handling in PostgresToGcs operator
URL: https://github.com/apache/airflow/pull/6572
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3489
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
According to the issue, `PostgresToGoogleCloudStorageOperator` had a problem
with handling postgres json data type in way that would allow to import the
exported file into BigQuery. So I've added a simple check and `json.dumps`.
I've checked if after that the data could be easily transferred to BQ (using
data transfer service) and everything worked. I've added also a system test.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[
https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973493#comment-16973493
]
ASF GitHub Bot commented on AIRFLOW-3489:
-
nuclearpinguin commented on pull request #6572: [AIRFLOW-3489] Improve json
data handling in PostgresToGcs operator
URL: https://github.com/apache/airflow/pull/6572
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3489
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
According to the issue, `PostgresToGoogleCloudStorageOperator` had a problem
with handling postgres json data type in way that would allow to import the
exported file into BigQuery. So I've added a simple check and `json.dumps`.
I've checked if after that the data could be easily transferred to BQ (using
data transfer service) and everything worked. I've added also a system test.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
> -
>
> Key: AIRFLOW-3489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3489
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Reporter: Duan Shiqiang
>Priority: Major
> Labels: bigquery
>
> PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb)
> as native python types (i.e. dictionary) to gcs new line separated json data.
> But it generates bigquery schema for that field as data type string which
> won't work if user want to import the gcs data into bigquery.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] GnunuX closed pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
GnunuX closed pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5904) "Trigger DAG" should not be redirected to the "tree" page but to the "dag_default_view" page
[ https://issues.apache.org/jira/browse/AIRFLOW-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973504#comment-16973504 ] ASF GitHub Bot commented on AIRFLOW-5904: - GnunuX commented on pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "Trigger DAG" should not be redirected to the "tree" page but to the > "dag_default_view" page > > > Key: AIRFLOW-5904 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5904 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.10.6 >Reporter: Emmanuel Garette >Priority: Minor > > When we are on the page of "DAG" and we click on "Trigger DAG", we are > redirected to the page "tree". > However, I changed the configuration element "dag_default_view" to "graph". > I think we should be redirected to the configured page name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) Add S3 to MySql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to MySql Operator (was: Add S3 to Mysql Operator) > Add S3 to MySql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) Add S3 to Mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to Mysql Operator (was: S3 to Mysql Operator) > Add S3 to Mysql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] GnunuX opened a new pull request #6573: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view" page
GnunuX opened a new pull request #6573: [AIRFLOW-5904] "Trigger DAG" should
redirect to the "dag_default_view" page
URL: https://github.com/apache/airflow/pull/6573
…" page
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Trigger DAG" should redirect to the "dag_default_view" page.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5904) "Trigger DAG" should not be redirected to the "tree" page but to the "dag_default_view" page
[
https://issues.apache.org/jira/browse/AIRFLOW-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973509#comment-16973509
]
ASF GitHub Bot commented on AIRFLOW-5904:
-
GnunuX commented on pull request #6573: [AIRFLOW-5904] "Trigger DAG" should
redirect to the "dag_default_view" page
URL: https://github.com/apache/airflow/pull/6573
…" page
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Trigger DAG" should redirect to the "dag_default_view" page.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> "Trigger DAG" should not be redirected to the "tree" page but to the
> "dag_default_view" page
>
>
> Key: AIRFLOW-5904
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5904
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.6
>Reporter: Emmanuel Garette
>Priority: Minor
>
> When we are on the page of "DAG" and we click on "Trigger DAG", we are
> redirected to the page "tree".
> However, I changed the configuration element "dag_default_view" to "graph".
> I think we should be redirected to the configured page name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
feluelle opened a new pull request #6574: [AIRFLOW-5924] Automatically unify
bucket name and key in S3Hook
URL: https://github.com/apache/airflow/pull/6574
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5924
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- change provide_bucket_name to provide bucket name also for function with
keys
- refactoring
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973513#comment-16973513
]
ASF GitHub Bot commented on AIRFLOW-5924:
-
feluelle commented on pull request #6574: [AIRFLOW-5924] Automatically unify
bucket name and key in S3Hook
URL: https://github.com/apache/airflow/pull/6574
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5924
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- change provide_bucket_name to provide bucket name also for function with
keys
- refactoring
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Automatically unify bucket name and key in S3Hook
> -
>
> Key: AIRFLOW-5924
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5924
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook
feluelle opened a new pull request #6575: [AIRFLOW-5921] Add bulk_load_custom
to MySqlHook
URL: https://github.com/apache/airflow/pull/6575
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5921
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
[
https://issues.apache.org/jira/browse/AIRFLOW-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973522#comment-16973522
]
ASF GitHub Bot commented on AIRFLOW-5921:
-
feluelle commented on pull request #6575: [AIRFLOW-5921] Add bulk_load_custom
to MySqlHook
URL: https://github.com/apache/airflow/pull/6575
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5921
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add bulk_load_custom to MySqlHook
> -
>
> Key: AIRFLOW-5921
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5921
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
feluelle opened a new pull request #6576: [AIRFLOW-5922] Add option to specify
the mysql client library used in MySqlHook
URL: https://github.com/apache/airflow/pull/6576
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5922
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- add mysql-connector-python library
- fix pylint issues
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
[
https://issues.apache.org/jira/browse/AIRFLOW-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973523#comment-16973523
]
ASF GitHub Bot commented on AIRFLOW-5922:
-
feluelle commented on pull request #6576: [AIRFLOW-5922] Add option to specify
the mysql client library used in MySqlHook
URL: https://github.com/apache/airflow/pull/6576
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5922
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- add mysql-connector-python library
- fix pylint issues
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add option to specify the mysql client library used in MySqlHook
>
>
> Key: AIRFLOW-5922
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5922
> Project: Apache Airflow
> Issue Type: New Feature
> Components: dependencies, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
