[jira] [Commented] (AIRFLOW-1209) Hive Hook with beeline doesn't execute query if hql query doesn't have \n in the end
[
https://issues.apache.org/jira/browse/AIRFLOW-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777687#comment-16777687
]
jack commented on AIRFLOW-1209:
---
I don't think Airflow should maintain backwards compatibility for bugs in other
packages.
> Hive Hook with beeline doesn't execute query if hql query doesn't have \n in
> the end
>
>
> Key: AIRFLOW-1209
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1209
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hive_hooks
>Affects Versions: 1.8.0, 1.9.0
>Reporter: pralabhkumar
>Priority: Major
>
> Airflow Hive operator doesn't run the query if the query doesn't end with \n
> . I am using hive 1.2 beeline . If I used just below query
> hql="select count(*) from test;"
> in my HiveOpertor ,then the query doesn't run. Since beeline require \n at
> the end of the file .
> But can we make sure in hive_hooks.py ,we by default add \n ,and doesn't
> leave the same to the user .
> Code should change to
> hql = "USE {schema};\n{hql}\n".format(**locals())
> instead of
> hql = "USE {schema};\n{hql}".format(**locals())
> Though this issue is fixed by beeline
> https://issues.apache.org/jira/browse/HIVE-10541
> in later version
> But for back word compatibility can we do the change in hive_hooks.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3952) wrong format of the beeline hiveconf parameters in hive_hooks.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Liu updated AIRFLOW-3952:
Description:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"-hiveconf" should be "--hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
was:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"\-hiveconf``" should be "\-\-hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
> wrong format of the beeline hiveconf parameters in hive_hooks.py
>
>
> Key: AIRFLOW-3952
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3952
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.8.0, 1.8.0b2, 1.8.0b3, 1.8.0b5, 1.8.0rc2, 1.8.0rc3,
> 1.8.0rc4, 1.8.1rc0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2
>Reporter: Yu Liu
>Assignee: Yu Liu
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
> Attachments: changed_all__-hiveconf__to__--hiveconf__.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Problem:
> The support to beeline cli parameters was introduced from 1.8.0, but the
> format of "hiveconf" was wrong.
> Fix:
> "-hiveconf" should be "--hiveconf".
> Reasons:
> According to the Hive Beeline source code
> ([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
> {code:java}
> private static final String HIVE_CONF_PREFIX = "--hiveconf";
> {code}
> only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are
> omitted.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3952) wrong format of the beeline hiveconf parameters in hive_hooks.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Liu updated AIRFLOW-3952:
Description:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"\-hiveconf" should be "\-\-hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "\-\-hiveconf" can be accepted. Parameters come with "\-hiveconf" are
omitted.
was:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"-hiveconf" should be "--hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
> wrong format of the beeline hiveconf parameters in hive_hooks.py
>
>
> Key: AIRFLOW-3952
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3952
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.8.0, 1.8.0b2, 1.8.0b3, 1.8.0b5, 1.8.0rc2, 1.8.0rc3,
> 1.8.0rc4, 1.8.1rc0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2
>Reporter: Yu Liu
>Assignee: Yu Liu
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
> Attachments: changed_all__-hiveconf__to__--hiveconf__.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Problem:
> The support to beeline cli parameters was introduced from 1.8.0, but the
> format of "hiveconf" was wrong.
> Fix:
> "\-hiveconf" should be "\-\-hiveconf".
> Reasons:
> According to the Hive Beeline source code
> ([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
> {code:java}
> private static final String HIVE_CONF_PREFIX = "--hiveconf";
> {code}
> only "\-\-hiveconf" can be accepted. Parameters come with "\-hiveconf" are
> omitted.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3952) wrong format of the beeline hiveconf parameters in hive_hooks.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Liu updated AIRFLOW-3952:
Description:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"\-hiveconf``" should be "\-\-hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
was:
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"-hiveconf" should be "--hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
> wrong format of the beeline hiveconf parameters in hive_hooks.py
>
>
> Key: AIRFLOW-3952
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3952
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.8.0, 1.8.0b2, 1.8.0b3, 1.8.0b5, 1.8.0rc2, 1.8.0rc3,
> 1.8.0rc4, 1.8.1rc0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2
>Reporter: Yu Liu
>Assignee: Yu Liu
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
> Attachments: changed_all__-hiveconf__to__--hiveconf__.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Problem:
> The support to beeline cli parameters was introduced from 1.8.0, but the
> format of "hiveconf" was wrong.
> Fix:
> "\-hiveconf``" should be "\-\-hiveconf".
> Reasons:
> According to the Hive Beeline source code
> ([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
>
> {code:java}
> private static final String HIVE_CONF_PREFIX = "--hiveconf";
> {code}
> only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are
> omitted.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3957) Ensure CLI works with connexion API
[ https://issues.apache.org/jira/browse/AIRFLOW-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3957: Issue Type: Improvement (was: Bug) > Ensure CLI works with connexion API > --- > > Key: AIRFLOW-3957 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3957 > Project: Apache Airflow > Issue Type: Improvement > Components: api, cli >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Ensure all the existing CLI commands which access airflow through the API > function correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3957) Ensure CLI works with connexion API
Drew Sonne created AIRFLOW-3957: --- Summary: Ensure CLI works with connexion API Key: AIRFLOW-3957 URL: https://issues.apache.org/jira/browse/AIRFLOW-3957 Project: Apache Airflow Issue Type: Bug Components: api, cli Affects Versions: 1.10.2 Reporter: Drew Sonne Fix For: 1.10.3 Ensure all the existing CLI commands which access airflow through the API function correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3954) Ensure airflow.api.auth.backend.default works with connexion
[ https://issues.apache.org/jira/browse/AIRFLOW-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3954: Component/s: authentication > Ensure airflow.api.auth.backend.default works with connexion > > > Key: AIRFLOW-3954 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3954 > Project: Apache Airflow > Issue Type: Sub-task > Components: api, authentication >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Make sure all the existing test cases for the default auth method pass, and > perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3956) Ensure airflow.api.auth.backend.kerberos_auth works with connexion
[ https://issues.apache.org/jira/browse/AIRFLOW-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3956: Component/s: authentication > Ensure airflow.api.auth.backend.kerberos_auth works with connexion > -- > > Key: AIRFLOW-3956 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3956 > Project: Apache Airflow > Issue Type: Sub-task > Components: api, authentication >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Make sure all the existing test cases for the kerberos_auth auth method pass, > and perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3955) Ensure airflow.api.auth.backend.deny_all works with connexion
[ https://issues.apache.org/jira/browse/AIRFLOW-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3955: Component/s: authentication > Ensure airflow.api.auth.backend.deny_all works with connexion > - > > Key: AIRFLOW-3955 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3955 > Project: Apache Airflow > Issue Type: Sub-task > Components: api, authentication >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Make sure all the existing test cases for the deny_all auth method pass, and > perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3954) Ensure airflow.api.auth.backend.default works with connexion
[
https://issues.apache.org/jira/browse/AIRFLOW-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Drew Sonne updated AIRFLOW-3954:
Summary: Ensure airflow.api.auth.backend.default works with connexion
(was: Ensure {{airflow.api.auth.backend.default}} works with connexion)
> Ensure airflow.api.auth.backend.default works with connexion
>
>
> Key: AIRFLOW-3954
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3954
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: api
>Affects Versions: 1.10.2
>Reporter: Drew Sonne
>Priority: Major
> Labels: aip-13, openapi
> Fix For: 1.10.3
>
>
> Make sure all the existing test cases for the default auth method pass, and
> perform some smoke tests to ensure the authentication works in practice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3956) Ensure airflow.api.auth.backend.kerberos_auth works with connexion
[ https://issues.apache.org/jira/browse/AIRFLOW-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3956: Description: Make sure all the existing test cases for the kerberos_auth auth method pass, and perform some smoke tests to ensure the authentication works in practice. (was: Make sure all the existing test cases for the deny_all auth method pass, and perform some smoke tests to ensure the authentication works in practice.) > Ensure airflow.api.auth.backend.kerberos_auth works with connexion > -- > > Key: AIRFLOW-3956 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3956 > Project: Apache Airflow > Issue Type: Sub-task > Components: api >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Make sure all the existing test cases for the kerberos_auth auth method pass, > and perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3955) Ensure airflow.api.auth.backend.deny_all works with connexion
Drew Sonne created AIRFLOW-3955: --- Summary: Ensure airflow.api.auth.backend.deny_all works with connexion Key: AIRFLOW-3955 URL: https://issues.apache.org/jira/browse/AIRFLOW-3955 Project: Apache Airflow Issue Type: Sub-task Components: api Affects Versions: 1.10.2 Reporter: Drew Sonne Fix For: 1.10.3 Make sure all the existing test cases for the default auth method pass, and perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3955) Ensure airflow.api.auth.backend.deny_all works with connexion
[ https://issues.apache.org/jira/browse/AIRFLOW-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew Sonne updated AIRFLOW-3955: Description: Make sure all the existing test cases for the deny_all auth method pass, and perform some smoke tests to ensure the authentication works in practice. (was: Make sure all the existing test cases for the default auth method pass, and perform some smoke tests to ensure the authentication works in practice.) > Ensure airflow.api.auth.backend.deny_all works with connexion > - > > Key: AIRFLOW-3955 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3955 > Project: Apache Airflow > Issue Type: Sub-task > Components: api >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Priority: Major > Labels: aip-13, openapi > Fix For: 1.10.3 > > > Make sure all the existing test cases for the deny_all auth method pass, and > perform some smoke tests to ensure the authentication works in practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3954) Ensure {{airflow.api.auth.backend.default}} works with connexion
Drew Sonne created AIRFLOW-3954:
---
Summary: Ensure {{airflow.api.auth.backend.default}} works with
connexion
Key: AIRFLOW-3954
URL: https://issues.apache.org/jira/browse/AIRFLOW-3954
Project: Apache Airflow
Issue Type: Sub-task
Components: api
Affects Versions: 1.10.2
Reporter: Drew Sonne
Fix For: 1.10.3
Make sure all the existing test cases for the default auth method pass, and
perform some smoke tests to ensure the authentication works in practice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3953) Refactor API route handlers to use Connexion
[
https://issues.apache.org/jira/browse/AIRFLOW-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Drew Sonne updated AIRFLOW-3953:
Description:
All existing routes under {{/api}} (eg, in
{{airflow/www/api/experimental/endpoints}} should be refactored to be handler
by the [connexion|https://connexion.readthedocs.io/en/latest/] library.
Where possible this should be done by affecting the existing codebase as little
as possible. There is some bridging code already written to handle this.
https://github.com/apache/airflow/pull/4640/files#diff-859f51803a6d3ee3746285e5abe016bbR90
was:
All existing routes under {{/api}} (eg, in
{{airflow/www/api/experimental/endpoints}} should be refactored to be handler
by the connexion library.
Where possible this should be done by affecting the existing codebase as little
as possible. There is some bridging code already written to handle this.
https://github.com/apache/airflow/pull/4640/files#diff-859f51803a6d3ee3746285e5abe016bbR90
> Refactor API route handlers to use Connexion
>
>
> Key: AIRFLOW-3953
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3953
> Project: Apache Airflow
> Issue Type: Improvement
> Components: api
>Affects Versions: 1.10.2
>Reporter: Drew Sonne
>Priority: Major
> Labels: aip-13, openapi
> Fix For: 1.10.3
>
>
> All existing routes under {{/api}} (eg, in
> {{airflow/www/api/experimental/endpoints}} should be refactored to be handler
> by the [connexion|https://connexion.readthedocs.io/en/latest/] library.
> Where possible this should be done by affecting the existing codebase as
> little as possible. There is some bridging code already written to handle
> this.
> https://github.com/apache/airflow/pull/4640/files#diff-859f51803a6d3ee3746285e5abe016bbR90
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3953) Refactor API route handlers to use Connexion
Drew Sonne created AIRFLOW-3953:
---
Summary: Refactor API route handlers to use Connexion
Key: AIRFLOW-3953
URL: https://issues.apache.org/jira/browse/AIRFLOW-3953
Project: Apache Airflow
Issue Type: Improvement
Components: api
Affects Versions: 1.10.2
Reporter: Drew Sonne
Fix For: 1.10.3
All existing routes under {{/api}} (eg, in
{{airflow/www/api/experimental/endpoints}} should be refactored to be handler
by the connexion library.
Where possible this should be done by affecting the existing codebase as little
as possible. There is some bridging code already written to handle this.
https://github.com/apache/airflow/pull/4640/files#diff-859f51803a6d3ee3746285e5abe016bbR90
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] KevinYang21 commented on issue #4396: [AIRFLOW-3585] - Add edges to database
KevinYang21 commented on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467314977 @ashb I think he meant SimpleDag. Since we already use such serialized representation to communicate between processes now, might also wanna do that for cross-component communication. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3952) wrong format of the beeline hiveconf parameters in hive_hooks.py
Yu Liu created AIRFLOW-3952:
---
Summary: wrong format of the beeline hiveconf parameters in
hive_hooks.py
Key: AIRFLOW-3952
URL: https://issues.apache.org/jira/browse/AIRFLOW-3952
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.2, 1.10.1, 1.10.0, 1.9.0, 1.8.2, 1.8.1, 1.8.1rc0,
1.8.0rc4, 1.8.0rc3, 1.8.0rc2, 1.8.0b5, 1.8.0b3, 1.8.0b2, 1.8.0
Reporter: Yu Liu
Assignee: Yu Liu
Fix For: 2.0.0, 1.10.3
Attachments: changed_all__-hiveconf__to__--hiveconf__.patch
Problem:
The support to beeline cli parameters was introduced from 1.8.0, but the format
of "hiveconf" was wrong.
Fix:
"-hiveconf" should be "--hiveconf".
Reasons:
According to the Hive Beeline source code
([link|https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java]),
{code:java}
private static final String HIVE_CONF_PREFIX = "--hiveconf";
{code}
only "--hiveconf" can be accepted. Parameters come with "-hiveconf" are omitted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory
feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory URL: https://github.com/apache/airflow/pull/4763#discussion_r260139573 ## File path: airflow/utils/file.py ## @@ -23,11 +23,15 @@ import errno import os import shutil +from contextlib import contextmanager Review comment: There is an ongoing AIP(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-6+Enforce+the+usage+of+a+code+formatter) which should provide a solution to the import order issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory
feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory URL: https://github.com/apache/airflow/pull/4763#discussion_r260139573 ## File path: airflow/utils/file.py ## @@ -23,11 +23,15 @@ import errno import os import shutil +from contextlib import contextmanager Review comment: there is an ongoing AIP(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-6+Enforce+the+usage+of+a+code+formatter) which should provide a solution to the import order issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory
feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory URL: https://github.com/apache/airflow/pull/4763#discussion_r260139573 ## File path: airflow/utils/file.py ## @@ -23,11 +23,15 @@ import errno import os import shutil +from contextlib import contextmanager Review comment: there is an ongoing AIP(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-6+Enforce+the+usage+of+a+code+formatter) which should provide a solution to import order issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef…
feng-tao commented on issue #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef… URL: https://github.com/apache/airflow/pull/4772#issuecomment-467303188 PTAL @dimberman This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3733) raise NameError if hql result had no record
[ https://issues.apache.org/jira/browse/AIRFLOW-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-3733. Fix > raise NameError if hql result had no record > --- > > Key: AIRFLOW-3733 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3733 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Affects Versions: 1.10.1 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: easyfix > Fix For: 1.10.3 > > > [to_csv|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/hive_hooks.py#L881-L903] > function will raise `NameError` while hql result had not record, we should > init variable i like > [oracle_hooks|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/oracle_hook.py#L92-L124] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3875) simplify SlackWebhookHook code and change docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-3875: - Fix Version/s: (was: 2.0.0) 1.10.3 > simplify SlackWebhookHook code and change docstring > --- > > Key: AIRFLOW-3875 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3875 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: easyfix > Fix For: 1.10.3 > > > SlackWebhookHook __init__ function do not need self.http_conn_id, and > docstring missing some necessary str -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3741) oracle_hook add extra config
[
https://issues.apache.org/jira/browse/AIRFLOW-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3741.
> oracle_hook add extra config
>
>
> Key: AIRFLOW-3741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3741
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: features
> Fix For: 1.10.3
>
>
> oracle_hook get_conn block just like below
>
> {code:java}
> if dsn and sid and not service_name:
> dsn = cx_Oracle.makedsn(dsn, conn.port, sid)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> elif dsn and service_name and not sid:
> dsn = cx_Oracle.makedsn(dsn, conn.port, service_name=service_name)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> else:
> conn = cx_Oracle.connect(conn.login, conn.password, conn.host){code}
>
> oracle_hook only support default encoding, not UTF-8.
> In cx_oracle [official
> doc|https://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.connect]
> and [github issue|https://github.com/oracle/python-cx_Oracle/issues/157] we
> know how to set connection encoding. I think is necessary add this feature to
> oracle_hook
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3741) oracle_hook add extra config
[
https://issues.apache.org/jira/browse/AIRFLOW-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-3741:
-
Summary: oracle_hook add extra config (was: oracle_hook not support
specific encoding)
> oracle_hook add extra config
>
>
> Key: AIRFLOW-3741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3741
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: features
> Fix For: 1.10.3
>
>
> oracle_hook get_conn block just like below
>
> {code:java}
> if dsn and sid and not service_name:
> dsn = cx_Oracle.makedsn(dsn, conn.port, sid)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> elif dsn and service_name and not sid:
> dsn = cx_Oracle.makedsn(dsn, conn.port, service_name=service_name)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> else:
> conn = cx_Oracle.connect(conn.login, conn.password, conn.host){code}
>
> oracle_hook only support default encoding, not UTF-8.
> In cx_oracle [official
> doc|https://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.connect]
> and [github issue|https://github.com/oracle/python-cx_Oracle/issues/157] we
> know how to set connection encoding. I think is necessary add this feature to
> oracle_hook
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3734) function `load_file` will not run if param `partition` is None
[
https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-3734:
-
Fix Version/s: (was: 2.0.0)
1.10.3
> function `load_file` will not run if param `partition` is None
> --
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH OVERWRITE INTO TABLE \n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from
> `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3734) function `load_file` will not run if param `partition` is None
[
https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3734.
Fix
> function `load_file` will not run if param `partition` is None
> --
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH OVERWRITE INTO TABLE \n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from
> `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[ https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3950. --- Resolution: Fixed > Improve AirflowSecurityManager.update_admin_perm_view > - > > Key: AIRFLOW-3950 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The implementation of `AirflowSecurityManager.update_admin_perm_view` can be > simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[ https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777478#comment-16777478 ] ASF subversion and git services commented on AIRFLOW-3950: -- Commit 61aed43a3f37332d323d6209f106900876eefc4f in airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=61aed43 ] [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view (#4774) - simplify the implementation - improve the performance > Improve AirflowSecurityManager.update_admin_perm_view > - > > Key: AIRFLOW-3950 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The implementation of `AirflowSecurityManager.update_admin_perm_view` can be > simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[ https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777477#comment-16777477 ] ASF GitHub Bot commented on AIRFLOW-3950: - feng-tao commented on pull request #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view URL: https://github.com/apache/airflow/pull/4774 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve AirflowSecurityManager.update_admin_perm_view > - > > Key: AIRFLOW-3950 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The implementation of `AirflowSecurityManager.update_admin_perm_view` can be > simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao merged pull request #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view
feng-tao merged pull request #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view URL: https://github.com/apache/airflow/pull/4774 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
codecov-io edited a comment on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#issuecomment-467223945 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=h1) Report > Merging [#4770](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c50a85146373bafb0cbf86850f834d63bd4dede8?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4770 +/- ## == + Coverage 74.45% 74.64% +0.19% == Files 450 450 Lines 2897228988 +16 == + Hits2157021637 +67 + Misses 7402 7351 -51 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `84.61% <100%> (+84.61%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `92.92% <0%> (+0.03%)` | :arrow_up: | | [airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==) | `74.5% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=footer). Last update [c50a851...a3c683e](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
codecov-io edited a comment on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#issuecomment-467223945 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=h1) Report > Merging [#4770](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c50a85146373bafb0cbf86850f834d63bd4dede8?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4770 +/- ## == + Coverage 74.45% 74.64% +0.19% == Files 450 450 Lines 2897228988 +16 == + Hits2157021637 +67 + Misses 7402 7351 -51 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `84.61% <100%> (+84.61%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `92.92% <0%> (+0.03%)` | :arrow_up: | | [airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==) | `74.5% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=footer). Last update [c50a851...a3c683e](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3951) No easy way to get query_execution_id from AWSAthensOperator
Rob Dinoff created AIRFLOW-3951:
---
Summary: No easy way to get query_execution_id from
AWSAthensOperator
Key: AIRFLOW-3951
URL: https://issues.apache.org/jira/browse/AIRFLOW-3951
Project: Apache Airflow
Issue Type: Improvement
Components: aws, contrib, operators
Affects Versions: 1.10.2
Reporter: Rob Dinoff
There does not seem to be an easy way to get the query_execution_id from the
AWSAthensOperator. This should be pushed as an xcom variable. Need this so
query_execution_id can be used in the AthenaSensor.
The following should be added to the end of the AWSAthenaOperator.execute
{code:java}
context["task_instance"].xcom_push(key="query_execution_id",
value=self.query_execution_id){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ryanyuan commented on a change in pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
ryanyuan commented on a change in pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#discussion_r260090937 ## File path: airflow/contrib/operators/gcs_to_bq.py ## @@ -24,6 +24,11 @@ from airflow.models import BaseOperator from airflow.utils.decorators import apply_defaults +try: +basestring Review comment: @mik-laj Thanks for the suggestion. I've updated my code. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4591: [AIRFLOW-3615][WIP] Parse hostname using netloc
codecov-io commented on issue #4591: [AIRFLOW-3615][WIP] Parse hostname using netloc URL: https://github.com/apache/airflow/pull/4591#issuecomment-467251879 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=h1) Report > Merging [#4591](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c552996483f5aab5e31793317b26aba62c13332e?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4591 +/- ## == + Coverage 74.44% 74.45% +<.01% == Files 450 450 Lines 2897428983 +9 == + Hits2157021579 +9 Misses 7404 7404 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4591/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `67.05% <100%> (+1.8%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=footer). Last update [c552996...530c2bf](https://codecov.io/gh/apache/airflow/pull/4591?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhongjiajie commented on issue #4773: [AIRFLOW-3767] Correct bulk insert function
zhongjiajie commented on issue #4773: [AIRFLOW-3767] Correct bulk insert function URL: https://github.com/apache/airflow/pull/4773#issuecomment-467251322 @ashb @Fokko PTAL, Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj commented on a change in pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
mik-laj commented on a change in pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#discussion_r260079130 ## File path: airflow/contrib/operators/gcs_to_bq.py ## @@ -24,6 +24,11 @@ from airflow.models import BaseOperator from airflow.utils.decorators import apply_defaults +try: +basestring Review comment: Is there any reason why you have not used the `six` package here? Using this package makes it easier to track where we have the code for the older version of Python. We plan to drop support for the older version, so we will simply delete all references to the `six` package. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
codecov-io commented on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#issuecomment-467223946 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=h1) Report > Merging [#4770](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c50a85146373bafb0cbf86850f834d63bd4dede8?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4770 +/- ## == + Coverage 74.45% 74.64% +0.19% == Files 450 450 Lines 2897228991 +19 == + Hits2157021640 +70 + Misses 7402 7351 -51 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `85.18% <100%> (+85.18%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `92.92% <0%> (+0.03%)` | :arrow_up: | | [airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==) | `74.5% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=footer). Last update [c50a851...35f7eb6](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
codecov-io commented on issue #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770#issuecomment-467223945 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=h1) Report > Merging [#4770](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c50a85146373bafb0cbf86850f834d63bd4dede8?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4770 +/- ## == + Coverage 74.45% 74.64% +0.19% == Files 450 450 Lines 2897228991 +19 == + Hits2157021640 +70 + Misses 7402 7351 -51 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fYnEucHk=) | `85.18% <100%> (+85.18%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `92.92% <0%> (+0.03%)` | :arrow_up: | | [airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/4770/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==) | `74.5% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=footer). Last update [c50a851...35f7eb6](https://codecov.io/gh/apache/airflow/pull/4770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3929) Modal buttons should be links
[ https://issues.apache.org/jira/browse/AIRFLOW-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777371#comment-16777371 ] ASF GitHub Bot commented on AIRFLOW-3929: - ashb commented on pull request #4742: [AIRFLOW-3929] Use anchor tags for modal links. URL: https://github.com/apache/airflow/pull/4742 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Modal buttons should be links > - > > Key: AIRFLOW-3929 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3929 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > > Buttons in the dag view modal aren't links, so users can't copy the url they > refer to, open the url in a new tab, etc. Instead, we change > `window.location` when a user clicks the button. We should use tags here > instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb merged pull request #4742: [AIRFLOW-3929] Use anchor tags for modal links.
ashb merged pull request #4742: [AIRFLOW-3929] Use anchor tags for modal links. URL: https://github.com/apache/airflow/pull/4742 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3929) Modal buttons should be links
[ https://issues.apache.org/jira/browse/AIRFLOW-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777372#comment-16777372 ] ASF subversion and git services commented on AIRFLOW-3929: -- Commit c552996483f5aab5e31793317b26aba62c13332e in airflow's branch refs/heads/master from Joshua Carp [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c552996 ] [AIRFLOW-3929] Use anchor tags for modal links on dag detail pages. (#4742) > Modal buttons should be links > - > > Key: AIRFLOW-3929 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3929 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > > Buttons in the dag view modal aren't links, so users can't copy the url they > refer to, open the url in a new tab, etc. Instead, we change > `window.location` when a user clicks the button. We should use tags here > instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3948) GCSToBQOperator list as templating field issue
[
https://issues.apache.org/jira/browse/AIRFLOW-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777355#comment-16777355
]
ASF GitHub Bot commented on AIRFLOW-3948:
-
ryanyuan commented on pull request #4770: [AIRFLOW-3948] GCSToBQOperator list
as templating field issue
URL: https://github.com/apache/airflow/pull/4770
Fix GCSToBQOp list templating
Add unit tests for GCSToBQOp
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-3948](https://issues.apache.org/jira/browse/AIRFLOW-3948) issues and
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
tests/contrib/operators/test_gcs_to_bq_operator.py
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> GCSToBQOperator list as templating field issue
> --
>
> Key: AIRFLOW-3948
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3948
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Ryan Yuan
>Assignee: Ryan Yuan
>Priority: Major
>
> With GoogleCloudStorageToBigQueryOperator's templating fields,
> `source_objects` and `schema_fields`, if parsed in values are lists, they
> will be recognized as strings, causing error returned by GCS.
> e.g.
> Variable before templating, source_objects = ['file_1.csv', 'file_2.csv']
> Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ryanyuan opened a new pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
ryanyuan opened a new pull request #4770: [AIRFLOW-3948] GCSToBQOperator list
as templating field issue
URL: https://github.com/apache/airflow/pull/4770
Fix GCSToBQOp list templating
Add unit tests for GCSToBQOp
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-3948](https://issues.apache.org/jira/browse/AIRFLOW-3948) issues and
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
tests/contrib/operators/test_gcs_to_bq_operator.py
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] dmateusp opened a new pull request #4777: Airflow 3918 add git sync ssh auth
dmateusp opened a new pull request #4777: Airflow 3918 add git sync ssh auth
URL: https://github.com/apache/airflow/pull/4777
Make sure you have checked _all_ steps below.
### Jira
> - [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
> - https://issues.apache.org/jira/browse/AIRFLOW-XXX
> - In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
https://issues.apache.org/jira/browse/AIRFLOW-3918
### Description
> - [X] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds support for Git Sync authentication through SSH key (e.g. a
GitHub deployment read-only key)
### Tests
> - [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
* test_worker_configuration_auth_both_ssh_and_user: tests that configuration
validation throws an exception when both user authentication and SSH
authentication are configured
* test_init_environment_using_git_sync_ssh: tests that the environment is
correctly set up for the init pod when using SSH auth
* test_make_pod_git_sync_ssh: tests that the make_pod function correctly
generates the config for the pod, covers security_context test
Additionally tested at HBC on our cluster
### Commits
> - [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
> 1. Subject is separated from body by a blank line
> 1. Subject is limited to 50 characters (not including Jira issue
reference)
> 1. Subject does not end with a period
> 1. Subject uses the imperative mood ("add", not "adding")
> 1. Body wraps at 72 characters
> 1. Body explains "what" and "why", not "how"
### Documentation
> - [X] In case of new functionality, my PR adds documentation that
describes how to use it.
> - When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
> - All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
> - [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database
ashb edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467199162 > 1. There is already a serialized representation of DAGs so DB serialization should probably go through that even though that might be a bit more work. @aoen What do you mean by this point? I don't see how security factors in to this PR/AIP at all - it doesn't change anything fundamental about how Airflow currently works, and is no better or worse for multi-tenancy than Variables, connections, or workers speaking directly to the DB anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database
ashb edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467199162 > 1. There is already a serialized representation of DAGs so DB serialization should probably go through that even though that might be a bit more work. @aoen What do you mean by this point? > reference to some encapsulation of all of the python dependencies for the DAG This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #4396: [AIRFLOW-3585] - Add edges to database
ashb commented on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467199162 @aoen What do you mean by this point? > 1. There is already a serialized representation of DAGs so DB serialization should probably go through that even though that might be a bit more work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ybendana commented on issue #4291: [AIRFLOW-1488] Add the DagRunSensor operator
ybendana commented on issue #4291: [AIRFLOW-1488] Add the DagRunSensor operator URL: https://github.com/apache/airflow/pull/4291#issuecomment-467130551 Hi @XD-DENG , yes that is correct. So, if I changed the name to TriggeredDagRunSensor it would be clearer what the functionality is? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] samuelwbock commented on a change in pull request #4675: [AIRFLOW-3853] Default to delete local logs after remote upload
samuelwbock commented on a change in pull request #4675: [AIRFLOW-3853] Default to delete local logs after remote upload URL: https://github.com/apache/airflow/pull/4675#discussion_r259958453 ## File path: airflow/utils/log/s3_task_handler.py ## @@ -84,7 +85,10 @@ def close(self): # read log and remove old logs to get just the latest additions with open(local_loc, 'r') as logfile: log = logfile.read() -self.s3_write(log, remote_loc) +successfully_written = self.s3_write(log, remote_loc) +# If the file was written out, remove it to avoid duplicating logs in S3 +if successfully_written: Review comment: Hey friends, changes made. Any other changes desired? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3947. --- Resolution: Fixed > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777141#comment-16777141 ] ASF subversion and git services commented on AIRFLOW-3947: -- Commit bb91246cfc6f7905c84aea4a44c11aa057b8f42c in airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bb91246 ] [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767) * [AIRFLOW-3947] Flash msg for no DAG-level access error It will show and remind user when a user clicks on a DAG that he/she doesn't have can_dag_read or can_dag_edit permissions. * Change the flash msg contents > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory
feng-tao commented on a change in pull request #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory URL: https://github.com/apache/airflow/pull/4763#discussion_r259945397 ## File path: airflow/utils/file.py ## @@ -23,11 +23,15 @@ import errno import os import shutil +from contextlib import contextmanager Review comment: agree with @ashb , the import is sorted based on module alphabetical order. Initially I thought you said contextmanager is considered as 3rd party. If they all belong to stdlib, @ashb's way should be the way to change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777139#comment-16777139 ] ASF GitHub Bot commented on AIRFLOW-3947: - feng-tao commented on pull request #4767: [AIRFLOW-3947] Flash msg for no DAG-level access error URL: https://github.com/apache/airflow/pull/4767 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777143#comment-16777143 ] ASF subversion and git services commented on AIRFLOW-3947: -- Commit bb91246cfc6f7905c84aea4a44c11aa057b8f42c in airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bb91246 ] [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767) * [AIRFLOW-3947] Flash msg for no DAG-level access error It will show and remind user when a user clicks on a DAG that he/she doesn't have can_dag_read or can_dag_edit permissions. * Change the flash msg contents > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777142#comment-16777142 ] ASF subversion and git services commented on AIRFLOW-3947: -- Commit bb91246cfc6f7905c84aea4a44c11aa057b8f42c in airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bb91246 ] [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767) * [AIRFLOW-3947] Flash msg for no DAG-level access error It will show and remind user when a user clicks on a DAG that he/she doesn't have can_dag_read or can_dag_edit permissions. * Change the flash msg contents > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777140#comment-16777140 ] ASF subversion and git services commented on AIRFLOW-3947: -- Commit bb91246cfc6f7905c84aea4a44c11aa057b8f42c in airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bb91246 ] [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767) * [AIRFLOW-3947] Flash msg for no DAG-level access error It will show and remind user when a user clicks on a DAG that he/she doesn't have can_dag_read or can_dag_edit permissions. * Change the flash msg contents > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao merged pull request #4767: [AIRFLOW-3947] Flash msg for no DAG-level access error
feng-tao merged pull request #4767: [AIRFLOW-3947] Flash msg for no DAG-level access error URL: https://github.com/apache/airflow/pull/4767 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4767: [AIRFLOW-3947] Flash msg for no DAG-level access error
feng-tao commented on issue #4767: [AIRFLOW-3947] Flash msg for no DAG-level access error URL: https://github.com/apache/airflow/pull/4767#issuecomment-467109337 thanks This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj closed pull request #4776: [3/3] Docs changes - Refactor base gcp hook
mik-laj closed pull request #4776: [3/3] Docs changes - Refactor base gcp hook URL: https://github.com/apache/airflow/pull/4776 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj closed pull request #4775: [1/3] Auto API
mik-laj closed pull request #4775: [1/3] Auto API URL: https://github.com/apache/airflow/pull/4775 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj opened a new pull request #4776: [3/3] Docs changes - Refactor base gcp hook
mik-laj opened a new pull request #4776: [3/3] Docs changes - Refactor base gcp
hook
URL: https://github.com/apache/airflow/pull/4776
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4769: [AIRFLOW-2511] Fix improper failed session commit handling
codecov-io edited a comment on issue #4769: [AIRFLOW-2511] Fix improper failed session commit handling URL: https://github.com/apache/airflow/pull/4769#issuecomment-466913688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=h1) Report > Merging [#4769](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/74b22337b45a1eb25585d52e35694e6b0eb81f03?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `78.66%`. [](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4769 +/- ## == + Coverage 74.44% 74.46% +0.01% == Files 450 450 Lines 2897328976 +3 == + Hits2157021577 +7 + Misses 7403 7399 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/4769/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `76.88% <78.66%> (+0.42%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=footer). Last update [74b2233...a63948a](https://codecov.io/gh/apache/airflow/pull/4769?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj opened a new pull request #4775: [1/3] Auto API
mik-laj opened a new pull request #4775: [1/3] Auto API
URL: https://github.com/apache/airflow/pull/4775
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jmcarp commented on issue #4742: [AIRFLOW-3929] Use anchor tags for modal links.
jmcarp commented on issue #4742: [AIRFLOW-3929] Use anchor tags for modal links. URL: https://github.com/apache/airflow/pull/4742#issuecomment-467076254 Ready for another look when you have time @mik-laj @ashb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aoen edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database
aoen edited a comment on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467068171 I love the spirit of this change, but I'm -1 for now until we come up with some kind of long term plan. Couple of high-level comments after skimming the PR: 1. There is already a serialized representation of DAGs so DB serialization should probably go through that even though that might be a bit more work. 2. There is currently a DB call for each DAG in your code, these should be batched as much as possible (just need to be careful that query doesn't get too big). Along this vein probably need to stress test this with a large number of large DAGs and multiply results by appropriate factor if the DB used for testing is local instead of remote to see what performance would be like for real-world usage. In the long run I envision DAG serialization happening on new Airflow clients, which send a request to a new Airflow service which basically serves as a CRUD wrapper around a DB to store both the SimpleDag as well as some kind of reference to some encapsulation of all of the python dependencies for the DAG (e.g. docker image name). This way all 3 of the webserver/worker/scheduler could use the same data model and source of truth. I feel it might make sense to figure out the long term plan first via an AIP and some brainstorming sessions and make sure there is an easy path forward from any intermediate proposals so we don't make our lives harder later undoing changes/figuring out how to do migrations. Security is another thing to keep in mind when thinking about this problem too since this work would be required for multi-tenancy in Airflow. @KevinYang21 I know you have been looking at this problem recently as well so curious what you think especially about short term/long term solutions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aoen commented on issue #4396: [AIRFLOW-3585] - Add edges to database
aoen commented on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-467068171 Couple of high-level comments: 1. There is already a serialized representation of DAGs so DB serialization should probably go through that even though that might be a bit more work. 2. There is currently a DB call for each DAG in your code, these should be batched as much as possible (just need to be careful that query doesn't get too big). Along this vein probably need to stress test this with a large number of large DAGs and multiply results by appropriate factor if the DB used for testing is local instead of remote to see what performance would be like for real-world usage. In the long run I envision DAG serialization happening on new Airflow clients, which send a request to a new Airflow service which basically serves as a CRUD wrapper around a DB to store both the SimpleDag as well as some kind of reference to some encapsulation of all of the python dependencies for the DAG (e.g. docker image name). This way all 3 of the webserver/worker/scheduler could use the same data model and source of truth. I feel it might make sense to figure out the long term plan first via an AIP and some brainstorming sessions and make sure there is an easy path forward from any intermediate proposals so we don't make our lives harder later undoing changes/figuring out how to do migrations. Security is another thing to keep in mind when thinking about this problem too since this work would be required for multi-tenancy in Airflow. @KevinYang21 I know you have been looking at this problem recently as well so curious what you think especially about short term/long term solutions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4773: [AIRFLOW-3767] Correct bulk insert function
codecov-io commented on issue #4773: [AIRFLOW-3767] Correct bulk insert function URL: https://github.com/apache/airflow/pull/4773#issuecomment-467066997 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=h1) Report > Merging [#4773](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/74b22337b45a1eb25585d52e35694e6b0eb81f03?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4773 +/- ## == + Coverage 74.44% 74.45% +<.01% == Files 450 450 Lines 2897328980 +7 == + Hits2157021577 +7 Misses 7403 7403 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/oracle\_hook.py](https://codecov.io/gh/apache/airflow/pull/4773/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9vcmFjbGVfaG9vay5weQ==) | `95.96% <100%> (+0.24%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=footer). Last update [74b2233...ed239f5](https://codecov.io/gh/apache/airflow/pull/4773?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bbcbruno commented on issue #4554: [AIRFLOW-3729] Support env variables with pod runtime information in KubernetesPodOperator
bbcbruno commented on issue #4554: [AIRFLOW-3729] Support env variables with pod runtime information in KubernetesPodOperator URL: https://github.com/apache/airflow/pull/4554#issuecomment-467066890 @OmerJog I just rebased my branch with master and there is some problems that need to be fixed (not from my PR) ``` ./tests/core.py:959:53: F812 list comprehension redefines 'f' from line 933 ./tests/contrib/operators/test_cassandra_to_gcs_operator.py:82:33: F812 list comprehension redefines 'd' from line 74 ./airflow/jobs.py:1736:29: F812 list comprehension redefines 'dag' from line 1726 ./airflow/contrib/operators/cassandra_to_gcs.py:261:69: F812 list comprehension redefines 'name' from line 253 ./airflow/contrib/operators/cassandra_to_gcs.py:272:54: F812 list comprehension redefines 'name' from line 264 ./airflow/contrib/operators/cassandra_to_gcs.py:272:60: F812 list comprehension redefines 'value' from line 264 ./airflow/contrib/hooks/azure_data_lake_hook.py:67:16: F821 undefined name 'FileNotFoundError' ``` Should I fix this issues? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] fenglu-g commented on issue #4769: [AIRFLOW-2511] Fix improper failed session commit handling
fenglu-g commented on issue #4769: [AIRFLOW-2511] Fix improper failed session commit handling URL: https://github.com/apache/airflow/pull/4769#issuecomment-467053724 @ashb thanks for the review, replied your comment and fixed the typo [/facepalm]. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view
XD-DENG commented on issue #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view URL: https://github.com/apache/airflow/pull/4774#issuecomment-467041403 @ashb @feng-tao PTAL. A low-hanging improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view
XD-DENG commented on issue #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view URL: https://github.com/apache/airflow/pull/4774#issuecomment-467040628 One test failed, likely due to transient/flaky error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259857508 ## File path: airflow/contrib/operators/gcp_translate_operator.py ## @@ -0,0 +1,121 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow import AirflowException +from airflow.contrib.hooks.gcp_translate_hook import CloudTranslateHook +from airflow.models import BaseOperator +from airflow.utils.decorators import apply_defaults + + +class CloudTranslateTextOperator(BaseOperator): +""" +Translate a string or list of strings. + +.. seealso:: +For more information on how to use this operator, take a look at the guide: +:ref:`howto/operator:CloudTranslateTextOperator` + +See https://cloud.google.com/translate/docs/translating-text + +Execute method returns str or list. + +This is a list of dictionaries for each queried value. Each +dictionary typically contains three keys (though not +all will be present in all cases). + +* ``detectedSourceLanguage``: The detected language (as an + ISO 639-1 language code) of the text. +* ``translatedText``: The translation of the text into the + target language. +* ``input``: The corresponding input value. +* ``model``: The model used to translate the text. + +If only a single value is passed, then only a single +dictionary is set as XCom return value. + +:type values: str or list +:param values: String or list of strings to translate. + +:type target_language: str +:param target_language: The language to translate results into. This + is required by the API and defaults to + the target language of the current instance. + +:type format_: str or None +:param format_: (Optional) One of ``text`` or ``html``, to specify + if the input text is plain text or HTML. + +:type source_language: str or None +:param source_language: (Optional) The language of the text to + be translated. + +:type customization_ids: str or list or None +:param customization_ids: (Optional) ID or list of customization IDs + for translation. Sets the ``cid`` parameter + in the query. + +:type model: str or None +:param model: (Optional) The model used to translate the text, such + as ``'base'`` or ``'nmt'``. + +""" + +# [START translate_template_fields] Review comment: oh... a bad review :joy: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259856068 ## File path: tests/contrib/hooks/test_gcp_translate_hook.py ## @@ -0,0 +1,80 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import unittest + +from airflow.contrib.hooks.gcp_translate_hook import CloudTranslateHook +from tests.contrib.utils.base_gcp_mock import mock_base_gcp_hook_default_project_id + +try: +from unittest import mock +except ImportError: +try: +import mock +except ImportError: +mock = None + +PROJECT_ID_TEST = 'project-id' + + Review comment: got This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
zhongjiajie commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259854768 ## File path: airflow/contrib/hooks/gcp_translate_hook.py ## @@ -0,0 +1,103 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from google.cloud.translate_v2 import Client +from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook + + +class CloudTranslateHook(GoogleCloudBaseHook): +""" +Hook for Google Cloud translate APIs. +""" + +_client = None + +def __init__(self, gcp_conn_id='google_cloud_default'): +super(CloudTranslateHook, self).__init__(gcp_conn_id) + +def get_conn(self): +""" +Retrieves connection to Cloud Translate + +:return: Google Cloud Translate client object. +:rtype: Client +""" +if not self._client: +self._client = Client(credentials=self._get_credentials()) +return self._client + +def translate( +self, values, target_language, format_=None, source_language=None, customization_ids=(), model=None +): +"""Translate a string or list of strings. + +See https://cloud.google.com/translate/docs/translating-text + +:type values: str or list +:param values: String or list of strings to translate. + +:type target_language: str +:param target_language: The language to translate results into. This +is required by the API and defaults to +the target language of the current instance. + +:type format_: str +:param format_: (Optional) One of ``text`` or ``html``, to specify +if the input text is plain text or HTML. + +:type source_language: str or None +:param source_language: (Optional) The language of the text to +be translated. + +:type customization_ids: str or list or None +:param customization_ids: (Optional) ID or list of customization IDs + for translation. Sets the ``cid`` parameter + in the query. + +:type model: str or None +:param model: (Optional) The model used to translate the text, such + as ``'base'`` or ``'nmt'``. + +:rtype: str or list +:returns: A list of dictionaries for each queried value. Each + dictionary typically contains three keys (though not + all will be present in all cases) + + * ``detectedSourceLanguage``: The detected language (as an +ISO 639-1 language code) of the text. + * ``translatedText``: The translation of the text into the +target language. + * ``input``: The corresponding input value. + * ``model``: The model used to translate the text. + + If only a single value is passed, then only a single + dictionary will be returned. +:raises: :class:`~exceptions.ValueError` if the number of + values and translations differ. +""" +client = self.get_conn() + Review comment: Ok, I get this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
codecov-io commented on issue #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#issuecomment-467036204 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=h1) Report > Merging [#4755](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/74b22337b45a1eb25585d52e35694e6b0eb81f03?src=pr=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `63.26%`. [](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4755 +/- ## == - Coverage 74.44% 74.42% -0.03% == Files 450 453 +3 Lines 2897329022 +49 == + Hits2157021600 +30 - Misses 7403 7422 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/contrib/example\_dags/example\_gcp\_translate.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcF90cmFuc2xhdGUucHk=) | `0% <0%> (ø)` | | | [airflow/contrib/hooks/gcp\_translate\_hook.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF90cmFuc2xhdGVfaG9vay5weQ==) | `69.23% <69.23%> (ø)` | | | [...irflow/contrib/operators/gcp\_translate\_operator.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNsYXRlX29wZXJhdG9yLnB5) | `84.61% <84.61%> (ø)` | | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.59% <0%> (-0.06%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=footer). Last update [74b2233...d55d31d](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
codecov-io edited a comment on issue #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#issuecomment-467036204 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=h1) Report > Merging [#4755](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/74b22337b45a1eb25585d52e35694e6b0eb81f03?src=pr=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `63.26%`. [](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4755 +/- ## == - Coverage 74.44% 74.42% -0.03% == Files 450 453 +3 Lines 2897329022 +49 == + Hits2157021600 +30 - Misses 7403 7422 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/contrib/example\_dags/example\_gcp\_translate.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcF90cmFuc2xhdGUucHk=) | `0% <0%> (ø)` | | | [airflow/contrib/hooks/gcp\_translate\_hook.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF90cmFuc2xhdGVfaG9vay5weQ==) | `69.23% <69.23%> (ø)` | | | [...irflow/contrib/operators/gcp\_translate\_operator.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNsYXRlX29wZXJhdG9yLnB5) | `84.61% <84.61%> (ø)` | | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4755/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.59% <0%> (-0.06%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=footer). Last update [74b2233...d55d31d](https://codecov.io/gh/apache/airflow/pull/4755?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2167) Scheduler's clear_nonexistent_import_errors function should be called on first iteration
[ https://issues.apache.org/jira/browse/AIRFLOW-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776940#comment-16776940 ] Kiran Pathak commented on AIRFLOW-2167: --- I like your solution. This bug is definitely a bit of frustrating :). > Scheduler's clear_nonexistent_import_errors function should be called on > first iteration > > > Key: AIRFLOW-2167 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2167 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Casey >Assignee: Casey >Priority: Minor > Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png > > > In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not > called until the amount of seconds defined by `dag_dir_list_interval` has > elapsed. If the scheduler is not alive for the duration of > `dag_dir_list_interval` (300 seconds) this cleanup never occurs. In some > environments this could result in error messages displaying on the UI > permanently, even if the DAG has been removed from the environment. > It was previously an Airflow best practice to have the scheduler run N > runtimes and terminate. Then, the scheduler would started again by an > auxiliary process like Docker or Supervisor. This situation is what brought > the bug to my attention. > My suggested fix is to tweak jobs.py to run the import error cleanup on the > first iteration and periodically as defined by `dag_dir_list_interval`. This > way, a scheduler setup with a small number of runs will still have old errors > cleaned up. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3819) k8s executor - Allow the configuration of a global default for pod resource request/limits
[
https://issues.apache.org/jira/browse/AIRFLOW-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
afusr closed AIRFLOW-3819.
--
Resolution: Workaround
This has be set at the namespace level within k8s.
For example:
{noformat}
apiVersion: v1
kind: LimitRange
metadata:
name: worker-pod-limit
spec:
limits:
- default:
memory: 1000Mi
cpu: 1
defaultRequest:
memory: 500Mi
cpu: 0.5
type: Container
{noformat}
Which sets default values for memory and cpu for any pods created by airflow.
> k8s executor - Allow the configuration of a global default for pod resource
> request/limits
> ---
>
> Key: AIRFLOW-3819
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3819
> Project: Apache Airflow
> Issue Type: Improvement
> Components: kubernetes
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
>
> Currently the kubernetes executor allows you to specify pod resources
> requests and limits (cpu and memory). For example:
> {noformat}
> # Limit resources on this operator/task with node affinity & tolerations
> three_task = PythonOperator(
> task_id="three_task", python_callable=print_stuff, dag=dag,
> executor_config={
> "KubernetesExecutor": {"request_memory": "128Mi",
>"limit_memory": "128Mi",
>"tolerations": tolerations,
>"affinity": affinity}}
> )
> {noformat}
> These values are used by kubernetes when making scaling decisions. It would
> be nice to be able to specify a global default for these values, to ensure
> that each pod airflow creates has a value specified for these properties.
> There is still the requirement to override these values on a dag by dag
> basis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3819) k8s executor - Allow the configuration of a global default for pod resource request/limits
[
https://issues.apache.org/jira/browse/AIRFLOW-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776924#comment-16776924
]
ASF GitHub Bot commented on AIRFLOW-3819:
-
atrbgithub commented on pull request #4660: [AIRFLOW-3819] - Allow the
configuration of a global default for work…
URL: https://github.com/apache/airflow/pull/4660
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> k8s executor - Allow the configuration of a global default for pod resource
> request/limits
> ---
>
> Key: AIRFLOW-3819
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3819
> Project: Apache Airflow
> Issue Type: Improvement
> Components: kubernetes
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
>
> Currently the kubernetes executor allows you to specify pod resources
> requests and limits (cpu and memory). For example:
> {noformat}
> # Limit resources on this operator/task with node affinity & tolerations
> three_task = PythonOperator(
> task_id="three_task", python_callable=print_stuff, dag=dag,
> executor_config={
> "KubernetesExecutor": {"request_memory": "128Mi",
>"limit_memory": "128Mi",
>"tolerations": tolerations,
>"affinity": affinity}}
> )
> {noformat}
> These values are used by kubernetes when making scaling decisions. It would
> be nice to be able to specify a global default for these values, to ensure
> that each pod airflow creates has a value specified for these properties.
> There is still the requirement to override these values on a dag by dag
> basis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] atrbgithub closed pull request #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work…
atrbgithub closed pull request #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work… URL: https://github.com/apache/airflow/pull/4660 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] atrbgithub commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work…
atrbgithub commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work… URL: https://github.com/apache/airflow/pull/4660#issuecomment-467028941 Going to close this. We tested with the above ( defaults set at the namespace level) which solves the problem outside of airflow. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[
https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776897#comment-16776897
]
ASF GitHub Bot commented on AIRFLOW-3950:
-
XD-DENG commented on pull request #4774: [AIRFLOW-3950] Improve
AirflowSecurityManager.update_admin_perm_view
URL: https://github.com/apache/airflow/pull/4774
### Jira
- https://issues.apache.org/jira/browse/AIRFLOW-3950
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR helps improve `AirflowSecurityManager.update_admin_perm_view` by
- Simplifying the implementation: avoid unnecessary for-loop by using set.
- Improving the performance (especially when there are many DAGs)
In terms of performance improvement, please refer to this simple toy
benchmarking:
```python
def fun1(a, b):
a=set(a)
for i in b:
if i not in a:
a.add(i)
list(a)
def fun2(a, b):
list(set(a) | set(b))
if __name__ == '__main__':
import timeit
print(timeit.timeit("fun1([1,2,3], range(100))", setup="from __main__
import fun1"))
print(timeit.timeit("fun2([1,2,3], range(100))", setup="from __main__
import fun2"))
```
**Result:**
```
12.91225419112
5.48773547994
[Finished in 18.5s]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve AirflowSecurityManager.update_admin_perm_view
> -
>
> Key: AIRFLOW-3950
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3950
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webserver
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> The implementation of `AirflowSecurityManager.update_admin_perm_view` can be
> simplified using set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #4774: [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view
XD-DENG opened a new pull request #4774: [AIRFLOW-3950] Improve
AirflowSecurityManager.update_admin_perm_view
URL: https://github.com/apache/airflow/pull/4774
### Jira
- https://issues.apache.org/jira/browse/AIRFLOW-3950
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR helps improve `AirflowSecurityManager.update_admin_perm_view` by
- Simplifying the implementation: avoid unnecessary for-loop by using set.
- Improving the performance (especially when there are many DAGs)
In terms of performance improvement, please refer to this simple toy
benchmarking:
```python
def fun1(a, b):
a=set(a)
for i in b:
if i not in a:
a.add(i)
list(a)
def fun2(a, b):
list(set(a) | set(b))
if __name__ == '__main__':
import timeit
print(timeit.timeit("fun1([1,2,3], range(100))", setup="from __main__
import fun1"))
print(timeit.timeit("fun2([1,2,3], range(100))", setup="from __main__
import fun2"))
```
**Result:**
```
12.91225419112
5.48773547994
[Finished in 18.5s]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[ https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3950: --- Summary: Improve AirflowSecurityManager.update_admin_perm_view (was: Simplify AirflowSecurityManager.update_admin_perm_view) > Improve AirflowSecurityManager.update_admin_perm_view > - > > Key: AIRFLOW-3950 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The implementation of `AirflowSecurityManager.update_admin_perm_view` can be > simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259826395 ## File path: setup.py ## @@ -180,6 +180,7 @@ def write_version(filename=os.path.join(*['airflow', 'google-cloud-bigtable==0.31.0', 'google-cloud-spanner>=1.7.1', 'google-cloud-vision>=0.35.2', +'google-cloud-translate>=1.3.3', Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3950) Simplify AirflowSecurityManager.update_admin_perm_view
Xiaodong DENG created AIRFLOW-3950: -- Summary: Simplify AirflowSecurityManager.update_admin_perm_view Key: AIRFLOW-3950 URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 Project: Apache Airflow Issue Type: Improvement Components: webserver Reporter: Xiaodong DENG Assignee: Xiaodong DENG The implementation of `AirflowSecurityManager.update_admin_perm_view` can be simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259825597 ## File path: airflow/contrib/hooks/gcp_translate_hook.py ## @@ -0,0 +1,103 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from google.cloud.translate_v2 import Client +from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook + + +class CloudTranslateHook(GoogleCloudBaseHook): +""" +Hook for Google Cloud translate APIs. +""" + +_client = None + +def __init__(self, gcp_conn_id='google_cloud_default'): +super(CloudTranslateHook, self).__init__(gcp_conn_id) + +def get_conn(self): +""" +Retrieves connection to Cloud Translate + +:return: Google Cloud Translate client object. +:rtype: Client +""" +if not self._client: +self._client = Client(credentials=self._get_credentials()) +return self._client + +def translate( +self, values, target_language, format_=None, source_language=None, customization_ids=(), model=None +): +"""Translate a string or list of strings. + +See https://cloud.google.com/translate/docs/translating-text + +:type values: str or list +:param values: String or list of strings to translate. + +:type target_language: str +:param target_language: The language to translate results into. This +is required by the API and defaults to +the target language of the current instance. + +:type format_: str +:param format_: (Optional) One of ``text`` or ``html``, to specify +if the input text is plain text or HTML. + +:type source_language: str or None +:param source_language: (Optional) The language of the text to +be translated. + +:type customization_ids: str or list or None +:param customization_ids: (Optional) ID or list of customization IDs + for translation. Sets the ``cid`` parameter + in the query. + +:type model: str or None +:param model: (Optional) The model used to translate the text, such + as ``'base'`` or ``'nmt'``. + +:rtype: str or list +:returns: A list of dictionaries for each queried value. Each + dictionary typically contains three keys (though not + all will be present in all cases) + + * ``detectedSourceLanguage``: The detected language (as an +ISO 639-1 language code) of the text. + * ``translatedText``: The translation of the text into the +target language. + * ``input``: The corresponding input value. + * ``model``: The model used to translate the text. + + If only a single value is passed, then only a single + dictionary will be returned. +:raises: :class:`~exceptions.ValueError` if the number of + values and translations differ. +""" +client = self.get_conn() + Review comment: We already have a debug log in the Operator. https://github.com/apache/airflow/pull/4755/files#diff-250e99f89f67f9e59efc842ed614fb59R116 which I used to see the results. And it makes sense to put it there usually - especially that sometimes several different hooks are used in single "execute" method of the operator and we usually add some logging there (as needed). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259823609 ## File path: airflow/contrib/operators/gcp_translate_operator.py ## @@ -0,0 +1,121 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow import AirflowException +from airflow.contrib.hooks.gcp_translate_hook import CloudTranslateHook +from airflow.models import BaseOperator +from airflow.utils.decorators import apply_defaults + + +class CloudTranslateTextOperator(BaseOperator): +""" +Translate a string or list of strings. + +.. seealso:: +For more information on how to use this operator, take a look at the guide: +:ref:`howto/operator:CloudTranslateTextOperator` + +See https://cloud.google.com/translate/docs/translating-text + +Execute method returns str or list. + +This is a list of dictionaries for each queried value. Each +dictionary typically contains three keys (though not +all will be present in all cases). + +* ``detectedSourceLanguage``: The detected language (as an + ISO 639-1 language code) of the text. +* ``translatedText``: The translation of the text into the + target language. +* ``input``: The corresponding input value. +* ``model``: The model used to translate the text. + +If only a single value is passed, then only a single +dictionary is set as XCom return value. + +:type values: str or list +:param values: String or list of strings to translate. + +:type target_language: str +:param target_language: The language to translate results into. This + is required by the API and defaults to + the target language of the current instance. + +:type format_: str or None +:param format_: (Optional) One of ``text`` or ``html``, to specify + if the input text is plain text or HTML. + +:type source_language: str or None +:param source_language: (Optional) The language of the text to + be translated. + +:type customization_ids: str or list or None +:param customization_ids: (Optional) ID or list of customization IDs + for translation. Sets the ``cid`` parameter + in the query. + +:type model: str or None +:param model: (Optional) The model used to translate the text, such + as ``'base'`` or ``'nmt'``. + +""" + +# [START translate_template_fields] Review comment: We are using those to generate user Sphinx-generated documentation automatically and include pieces of actual working code in the documentation. We are showing which fields are templated to the user of the operator. See https://github.com/apache/airflow/pull/4755/files#diff-40e5e831dc73c82a2aba583e5f89f8a8R2372 It's really useful as the documentation is rather up-to-date this way. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259822434 ## File path: tests/contrib/hooks/test_gcp_translate_hook.py ## @@ -0,0 +1,80 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import unittest + +from airflow.contrib.hooks.gcp_translate_hook import CloudTranslateHook +from tests.contrib.utils.base_gcp_mock import mock_base_gcp_hook_default_project_id + +try: +from unittest import mock +except ImportError: +try: +import mock +except ImportError: +mock = None + +PROJECT_ID_TEST = 'project-id' + + Review comment: This pattern is sprinkled all-over the Airflow code and copy everywhere. It's rather unlikely for mock to be None - in different python versions you simply expect it in either unittests or directly imported. I think this problem will be gone fairly soon when Airflow 2.0 drops python 2.7 compatibility, and until then I'd rather leave it as is. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator
potiuk commented on a change in pull request #4755: [AIRFLOW-3939] Add Google Cloud Translate operator URL: https://github.com/apache/airflow/pull/4755#discussion_r259820613 ## File path: setup.py ## @@ -180,6 +180,7 @@ def write_version(filename=os.path.join(*['airflow', 'google-cloud-bigtable==0.31.0', 'google-cloud-spanner>=1.7.1', 'google-cloud-vision>=0.35.2', +'google-cloud-translate>=1.3.3', Review comment: Good catch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3222) Remove bql keyword from BigQueryOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3222. Resolution: Fixed Fix Version/s: 2.0.0 > Remove bql keyword from BigQueryOperator > > > Key: AIRFLOW-3222 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3222 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-32) Remove deprecated features prior to releasing Airflow 2.0
[ https://issues.apache.org/jira/browse/AIRFLOW-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776852#comment-16776852 ] Ash Berlin-Taylor commented on AIRFLOW-32: -- There are probably more still - that list isn't necessarily exhaustive. Someone would need to go through the code in more detail > Remove deprecated features prior to releasing Airflow 2.0 > - > > Key: AIRFLOW-32 > URL: https://issues.apache.org/jira/browse/AIRFLOW-32 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Jeremiah Lowin >Priority: Major > Labels: deprecated > Fix For: 2.0.0 > > > A number of features have been marked for deprecation in Airflow 2.0. They > need to be deleted prior to release. > Usually the error message or comments will mention Airflow 2.0 with either a > #TODO or #FIXME. > Tracking list (not necessarily complete!): > JIRA: > AIRFLOW-31 > AIRFLOW-200 > GitHub: > https://github.com/airbnb/airflow/pull/1137/files#diff-1c2404a3a60f829127232842250ff406R233 > https://github.com/airbnb/airflow/pull/1219 > https://github.com/airbnb/airflow/pull/1285 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3767) oracle hook bulk insert error while target_fields is DEFAULT values
[
https://issues.apache.org/jira/browse/AIRFLOW-3767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776853#comment-16776853
]
ASF GitHub Bot commented on AIRFLOW-3767:
-
zhongjiajie commented on pull request #4773: [AIRFLOW-3767] Correct bulk insert
function
URL: https://github.com/apache/airflow/pull/4773
Fix Oracle hook bulk_insert bug when
`target_fields` is None or `rows` is
empty iterable
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3767
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Fix Oracle_hook bulk_insert_rows while param `target_fields` is None and
`rows` is empty iterable.
### Tests
- [x] My PR adds the following unit tests :
`test/hooks/test_oracle_hook.py` function
`test_bulk_insert_rows_without_fields` and `test_bulk_insert_rows_no_rows`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] not a new functionality
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> oracle hook bulk insert error while target_fields is DEFAULT values
> ---
>
> Key: AIRFLOW-3767
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3767
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix
>
> {code:java}
> def bulk_insert_rows(self, table, rows, target_fields=None,
> commit_every=5000):
> conn = self.get_conn()
> cursor = conn.cursor()
> values = ', '.join(':%s' % i for i in range(1, len(target_fields) + 1))
> prepared_stm = 'insert into {tablename} ({columns}) values
> ({values})'.format(
> tablename=table,
> columns=', '.join(target_fields),
> values=values,
> )
> {code}
> *target_fields* default *None* will leads
>
> {code:java}
> len(target_fields)
> {code}
> and
> {code:java}
> ', '.join(target_fields)
> {code}
> fail
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhongjiajie opened a new pull request #4773: [AIRFLOW-3767] Correct bulk insert function
zhongjiajie opened a new pull request #4773: [AIRFLOW-3767] Correct bulk insert
function
URL: https://github.com/apache/airflow/pull/4773
Fix Oracle hook bulk_insert bug when
`target_fields` is None or `rows` is
empty iterable
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3767
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Fix Oracle_hook bulk_insert_rows while param `target_fields` is None and
`rows` is empty iterable.
### Tests
- [x] My PR adds the following unit tests :
`test/hooks/test_oracle_hook.py` function
`test_bulk_insert_rows_without_fields` and `test_bulk_insert_rows_no_rows`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] not a new functionality
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-1630) Tasks do not end up in state UPSTREAM_FAILED consistently
[
https://issues.apache.org/jira/browse/AIRFLOW-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-1630.
Resolution: Fixed
Fix Version/s: 1.9.0
> Tasks do not end up in state UPSTREAM_FAILED consistently
> -
>
> Key: AIRFLOW-1630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1630
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.2
> Environment: Ubuntu 16.04, PostgreSQL 9.6.5, Python 2.7.12
>Reporter: Nikolay Petrachkov
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: Screen Shot 2017-09-21 at 17.36.37.png
>
>
> Given a simple DAG with 2 tasks: BashOperator and DummyOperator
> BashOperator has a command "exit 1"
> BashOperator is upstream for DummyOperator.
> When we run this DAG we expect BashOperator to fail and DummyOperator to be
> in state UPSTREAM_FAILED.
> Actual result: BashOperator is in state FAILED and DummyOperator is in state
> None.
> Code:
> {code:python}
> from airflow import DAG
> from datetime import datetime
> from airflow.operators.dummy_operator import DummyOperator
> from airflow.operators.bash_operator import BashOperator
> default_args = {
> 'owner': 'airflow',
> 'start_date': datetime(2017, 9, 20),
> 'retries': 0
> }
> dag = DAG(
> 'delivery-failed',
> default_args=default_args,
> schedule_interval=None
> )
> failed_bash = "exit 1"
> bash_task = BashOperator(
> task_id='bash-task',
> bash_command=failed_bash,
> dag=dag
> )
> end_task = DummyOperator(
> task_id='end',
> dag=dag
> )
> bash_task >> end_task
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3172) AttributeError: 'DagModel' object has no attribute 'execution_date'
[
https://issues.apache.org/jira/browse/AIRFLOW-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor closed AIRFLOW-3172.
--
Resolution: Duplicate
Fix Version/s: (was: 1.10.3)
Fixed elsewhere.
> AttributeError: 'DagModel' object has no attribute 'execution_date'
> ---
>
> Key: AIRFLOW-3172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3172
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Docker Environment: python:3.6-stretch
>Reporter: Vinnson Lee
>Priority: Major
>
> 2018-10-09 10:13:28,430] ERROR in app: Exception on /admin/dagmodel/ [GET]
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in
> reraise
> raise value
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368,
> in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1900, in index_view
> return_url=self._get_list_url(view_args),
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 308,
> in render
> return render_template(template, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 134,
> in render_template
> context, ctx.app)
> File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 116,
> in _render
> rv = template.render(context)
> File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line
> 989, in render
> return self.environment.handle_exception(exc_info, True)
> File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line
> 754, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in
> reraise
> raise value.with_traceback(tb)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 22, in top-level template code
> \{% import 'admin/actions.html' as actionlib with context %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
> line 18, in top-level template code
> \{% extends 'admin/base.html' %}
> File
> "/usr/local/lib/python3.6/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
> line 30, in top-level template code
> \{% block page_body %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
> line 107, in block "page_body"
> \{% block body %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 67, in block "body"
> \{% block model_list_table %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 115, in block "model_list_table"
> \{% block list_row scoped %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 143, in block "list_row"
> \{{ get_value(row, c) }}
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1742, in get_list_value
> self.column_type_formatters,
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1707, in _get_list_value
> value = column_fmt(self, context, model, name)
> File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line
> 124, in dag_link
> execution_date=m.execution_date)
> AttributeError: 'DagModel' object has no attribute 'execution_date'
>
>
> Its fine to work with SQLlite, but not with mysql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2715) Dataflow template operator dosenot support region parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-2715. Resolution: Fixed Fix Version/s: (was: 2.0.0) 1.10.3 > Dataflow template operator dosenot support region parameter > --- > > Key: AIRFLOW-2715 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2715 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.9.0 >Reporter: Mohammed Tameem >Priority: Critical > Fix For: 1.10.3 > > > The DataflowTemplateOperator uses dataflow.projects.templates.launch which > has a region parameter but only supports execution of the dataflow job in the > us-central1 region. Alternatively there is another api, > dataflow.projects.locations.templates.launch which supports execution of the > template in all regional endpoints provided by google cloud. > It would be great if, > # The base REST API of this operator could be changed from > "dataflow.projects.templates.launch" to > "dataflow.projects.locations.templates.launch" > # A templated region paramter was included in the operator to run the > dataflow job in the requested regional endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
