[jira] [Updated] (AIRFLOW-3504) Extend/refine the functionality of "/health" endpoint
[ https://issues.apache.org/jira/browse/AIRFLOW-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3504: --- Summary: Extend/refine the functionality of "/health" endpoint (was: Refine the functionality of "/health" endpoint) > Extend/refine the functionality of "/health" endpoint > - > > Key: AIRFLOW-3504 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3504 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > For both www/ and www_rbac/, there is "/health" endpoint, which is supposed > to help users know the "health" condition of the Airflow application. > But the functionality of it at this moment is very limited. It can only help > check if the webserver is running, but actually the "health" of the > application may be bad even though this endpoint returns "The server is > healthy!" (scheduler may be down, database backend may be disconnected, etc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3504) Refine the functionality of "/health" endpoint
Xiaodong DENG created AIRFLOW-3504: -- Summary: Refine the functionality of "/health" endpoint Key: AIRFLOW-3504 URL: https://issues.apache.org/jira/browse/AIRFLOW-3504 Project: Apache Airflow Issue Type: Improvement Components: webapp Reporter: Xiaodong DENG Assignee: Xiaodong DENG For both www/ and www_rbac/, there is "/health" endpoint, which is supposed to help users know the "health" condition of the Airflow application. But the functionality of it at this moment is very limited. It can only help check if the webserver is running, but actually the "health" of the application may be bad even though this endpoint returns "The server is healthy!" (scheduler may be down, database backend may be disconnected, etc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3479) Records of "Log" table should be kept (by default) when users delete a DAG.
Xiaodong DENG created AIRFLOW-3479: -- Summary: Records of "Log" table should be kept (by default) when users delete a DAG. Key: AIRFLOW-3479 URL: https://issues.apache.org/jira/browse/AIRFLOW-3479 Project: Apache Airflow Issue Type: Improvement Affects Versions: 1.10.1 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Currently when we delete a DAG (using API or from the UI), it will delete all related records in all tables (all tables in which "dag_id" is available), including "log" table. However, the records in "log" table should be kept (by default). This would be ideal for multiple reasons, like auditing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3432) Test is missing for feature "Delete DAG in UI"
[ https://issues.apache.org/jira/browse/AIRFLOW-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3432: --- Summary: Test is missing for feature "Delete DAG in UI" (was: Test is missing for [AIRFLOW-3233](PR 4069)) > Test is missing for feature "Delete DAG in UI" > -- > > Key: AIRFLOW-3432 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3432 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.2 > > > In [https://github.com/apache/incubator-airflow/pull/4069,] "delete DAG in > UI" feature is fixed, but no test was added for it. > > We should have test for it to release it with more confidence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3432) Test is missing for [AIRFLOW-3233](PR 4069)
Xiaodong DENG created AIRFLOW-3432: -- Summary: Test is missing for [AIRFLOW-3233](PR 4069) Key: AIRFLOW-3432 URL: https://issues.apache.org/jira/browse/AIRFLOW-3432 Project: Apache Airflow Issue Type: Test Components: tests Affects Versions: 1.10.1 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Fix For: 1.10.2 In [https://github.com/apache/incubator-airflow/pull/4069,] "delete DAG in UI" feature is fixed, but no test was added for it. We should have test for it to release it with more confidence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3323) Support Basic Authentication for Flower
[ https://issues.apache.org/jira/browse/AIRFLOW-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3323: --- Affects Version/s: 1.10.1 > Support Basic Authentication for Flower > --- > > Key: AIRFLOW-3323 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3323 > Project: Apache Airflow > Issue Type: New Feature > Components: celery >Affects Versions: 1.10.0, 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > The current `airflow flower` doesn't come with any authentication. This may > make essential information exposed to in an untrusted environment. > Currently Flower itself supports > * HTTP Basic Authentication > * Google OAuth 2.0 > * GitHub OAuth > Given Flower is not really the most essential component of Airflow, we don't > have to support all its authentication methods. But may be good to at least > support Basic Authentication. > > This ticket adds support to Basic Authentication for `Airflow Flower`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3325) Fix the DAGs page - column "Recent Tasks" display issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3325: --- Summary: Fix the DAGs page - column "Recent Tasks" display issue (was: Fix the DAGs page display issue) > Fix the DAGs page - column "Recent Tasks" display issue > --- > > Key: AIRFLOW-3325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3325 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: Screen Shot 2018-11-10 at 11.01.43 AM.png > > > In master branch and pre-release 1.10.1b1: > In the main UI page, the "Recent Tasks" column is not displayed completely > (the last circle "scheduled" is not displayed completely). > Reason behind: a new task state "Skipped" is added while UI was not adjusted > accordingly. > !Screen Shot 2018-11-10 at 11.01.43 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3325) Fix the DAGs page - column "Recent Tasks" display issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3325: --- Description: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordingly. Please check the screenshot attached. was: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordingly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! > Fix the DAGs page - column "Recent Tasks" display issue > --- > > Key: AIRFLOW-3325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3325 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: Screen Shot 2018-11-10 at 11.01.43 AM.png > > > In master branch and pre-release 1.10.1b1: > In the main UI page, the "Recent Tasks" column is not displayed completely > (the last circle "scheduled" is not displayed completely). > Reason behind: a new task state "Skipped" is added while UI was not adjusted > accordingly. > > Please check the screenshot attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3325) Fix the DAGs page display issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3325: --- Description: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordinly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! was: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordinly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! > Fix the DAGs page display issue > --- > > Key: AIRFLOW-3325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3325 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: Screen Shot 2018-11-10 at 11.01.43 AM.png > > > In master branch and pre-release 1.10.1b1: > In the main UI page, the "Recent Tasks" column is not displayed completely > (the last circle "scheduled" is not displayed completely). > Reason behind: a new task state "Skipped" is added while UI was not adjusted > accordinly. > !Screen Shot 2018-11-10 at 11.01.43 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3325) Fix the DAGs page display issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3325: --- Description: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordingly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! was: In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordinly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! > Fix the DAGs page display issue > --- > > Key: AIRFLOW-3325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3325 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: Screen Shot 2018-11-10 at 11.01.43 AM.png > > > In master branch and pre-release 1.10.1b1: > In the main UI page, the "Recent Tasks" column is not displayed completely > (the last circle "scheduled" is not displayed completely). > Reason behind: a new task state "Skipped" is added while UI was not adjusted > accordingly. > !Screen Shot 2018-11-10 at 11.01.43 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3325) Fix the DAGs page display issue
Xiaodong DENG created AIRFLOW-3325: -- Summary: Fix the DAGs page display issue Key: AIRFLOW-3325 URL: https://issues.apache.org/jira/browse/AIRFLOW-3325 Project: Apache Airflow Issue Type: Improvement Components: ui Affects Versions: 1.10.1 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Fix For: 1.10.1 Attachments: Screen Shot 2018-11-10 at 11.01.43 AM.png In master branch and pre-release 1.10.1b1: In the main UI page, the "Recent Tasks" column is not displayed completely (the last circle "scheduled" is not displayed completely). Reason behind: a new task state "Skipped" is added while UI was not adjusted accordinly. !Screen Shot 2018-11-10 at 11.01.43 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3323) Support Basic Authentication for Flower
[ https://issues.apache.org/jira/browse/AIRFLOW-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3323: --- Description: The current `airflow flower` doesn't come with any authentication. This may make essential information exposed to in an untrusted environment. Currently Flower itself supports * HTTP Basic Authentication * Google OAuth 2.0 * GitHub OAuth Given Flower is not really the most essential component of Airflow, we don't have to support all its authentication methods. But may be good to at least support Basic Authentication. This ticket adds support to Basic Authentication for `Airflow Flower`. was: The current `airflow flower` doesn't come with any authentication. This may make essential information exposed to in an untrusted environment. Currently Flower itself supports * HTTP Basic Authentication * Google OAuth 2.0 * GitHub OAuth Given Flower is not really the most essential component of Airflow, we don't have to support all its authentication methods. But may be good to at least support Basic Authentication. > Support Basic Authentication for Flower > --- > > Key: AIRFLOW-3323 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3323 > Project: Apache Airflow > Issue Type: New Feature > Components: celery >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The current `airflow flower` doesn't come with any authentication. This may > make essential information exposed to in an untrusted environment. > Currently Flower itself supports > * HTTP Basic Authentication > * Google OAuth 2.0 > * GitHub OAuth > Given Flower is not really the most essential component of Airflow, we don't > have to support all its authentication methods. But may be good to at least > support Basic Authentication. > > This ticket adds support to Basic Authentication for `Airflow Flower`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3323) Support Basic Authentication for Flower
Xiaodong DENG created AIRFLOW-3323: -- Summary: Support Basic Authentication for Flower Key: AIRFLOW-3323 URL: https://issues.apache.org/jira/browse/AIRFLOW-3323 Project: Apache Airflow Issue Type: New Feature Components: celery Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG The current `airflow flower` doesn't come with any authentication. This may make essential information exposed to in an untrusted environment. Currently Flower itself supports * HTTP Basic Authentication * Google OAuth 2.0 * GitHub OAuth Given Flower is not really the most essential component of Airflow, we don't have to support all its authentication methods. But may be good to at least support Basic Authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3301) Update CI test for [AIRFLOW-3132] (PR #3977)
Xiaodong DENG created AIRFLOW-3301: -- Summary: Update CI test for [AIRFLOW-3132] (PR #3977) Key: AIRFLOW-3301 URL: https://issues.apache.org/jira/browse/AIRFLOW-3301 Project: Apache Airflow Issue Type: Test Components: tests Reporter: Xiaodong DENG Assignee: Xiaodong DENG In PR [https://github.com/apache/incubator-airflow/pull/3977,] test is not updated accordingly, and it results in CI failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG edited comment on AIRFLOW-3239 at 11/4/18 2:54 PM: - Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] - tests/operators/operator.py - tests/operators/bash_operator.py - tests/jobs.py was (Author: xd-deng): Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] - tests/operators/operator.py - tests/operators/bash_operator.py > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG edited comment on AIRFLOW-3239 at 11/4/18 1:47 PM: - Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] - tests/operators/operator.py - tests/operators/bash_operator.py was (Author: xd-deng): Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] - tests/operators/operator.py > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG edited comment on AIRFLOW-3239 at 11/4/18 2:12 AM: - Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] - tests/operators/operator.py was (Author: xd-deng): Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3295) Potential Security Issue in DaskExecutor
Xiaodong DENG created AIRFLOW-3295: -- Summary: Potential Security Issue in DaskExecutor Key: AIRFLOW-3295 URL: https://issues.apache.org/jira/browse/AIRFLOW-3295 Project: Apache Airflow Issue Type: Bug Components: executor Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG When user decide to use SSL encryption for DaskExecutor communications, `Distributed.Security` object will be created. However, one argument (`require_encryption`) is missed to be set to `True` (its default value is `False`). This may fail the SSL encryption setting-up. Current implementation: [https://github.com/apache/incubator-airflow/blob/120f4856cdea5134971c4c4a239ddbfdc80db77e/airflow/executors/dask_executor.py#L48] Related Documentation: http://distributed.dask.org/en/latest/tls.html?highlight=require_encryption#parameters -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG edited comment on AIRFLOW-3239 at 10/28/18 10:47 AM: --- Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] - tests/[impersonation.py|https://github.com/XD-DENG/incubator-airflow/commit/8d806a88599806d7e3e28f7f3ebcacd3e71ee60d] was (Author: xd-deng): Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG edited comment on AIRFLOW-3239 at 10/28/18 10:47 AM: --- Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py - tests/[configuration.py|https://github.com/XD-DENG/incubator-airflow/commit/ca14afc273c326fa3c80da4a8e29e58c5f0c3489] was (Author: xd-deng): Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1278#comment-1278 ] Xiaodong DENG commented on AIRFLOW-3239: Opened branch [https://github.com/XD-DENG/incubator-airflow/tree/further_fix_test] to work on this issue. *Fixed tests (to be updated continuously)*: - tests/utils.py > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3175) Fix docstring format in airflow/jobs.py
[ https://issues.apache.org/jira/browse/AIRFLOW-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG resolved AIRFLOW-3175. Resolution: Fixed > Fix docstring format in airflow/jobs.py > --- > > Key: AIRFLOW-3175 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3175 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Trivial > > Some minor issues in docstring format in airflow/jobs.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
[ https://issues.apache.org/jira/browse/AIRFLOW-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658889#comment-16658889 ] Xiaodong DENG commented on AIRFLOW-3239: Hi [~ashb], thanks for checking on this as well. For - tests/operators/bash_operator.py - tests/operators/operator.py I'm aware of them, but got CI failures when I tried to simply rename them (prepend with "test_"), so haven't fixed them yet in my two earlier PRs. May continue on them later. If you have got the solution to fix them, kindly proceed. Cheers > Test discovery partial fails due to incorrect name of the test files > > > Key: AIRFLOW-3239 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed > the incorrect name of some test files (resulting in partial failure in test > discovery). > There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3239) Test discovery partial fails due to incorrect name of the test files
Xiaodong DENG created AIRFLOW-3239: -- Summary: Test discovery partial fails due to incorrect name of the test files Key: AIRFLOW-3239 URL: https://issues.apache.org/jira/browse/AIRFLOW-3239 Project: Apache Airflow Issue Type: Bug Components: tests Reporter: Xiaodong DENG Assignee: Xiaodong DENG In PR [https://github.com/apache/incubator-airflow/pull/4049,] I have fixed the incorrect name of some test files (resulting in partial failure in test discovery). There are some other scripts with this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3203) Bugs in DockerOperator & Some operator test scripts were named incorrectly
[ https://issues.apache.org/jira/browse/AIRFLOW-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3203: --- Summary: Bugs in DockerOperator & Some operator test scripts were named incorrectly (was: Bugs in DockerOperator & some operator tests) > Bugs in DockerOperator & Some operator test scripts were named incorrectly > -- > > Key: AIRFLOW-3203 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3203 > Project: Apache Airflow > Issue Type: Bug > Components: operators, tests >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based > on documentation of Python package "docker". > In addition, its test is not really working due to incorrect file name. This > also happens for some other test scripts for Operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3203) Bugs in DockerOperator & Some operator test scripts were named incorrectly
[ https://issues.apache.org/jira/browse/AIRFLOW-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3203: --- Description: Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based on documentation of Python package "docker". In addition, its test is not really working due to incorrect file name. This also happens for some other test scripts for Operators. This results in test discovery failure. was: Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based on documentation of Python package "docker". In addition, its test is not really working due to incorrect file name. This also happens for some other test scripts for Operators. > Bugs in DockerOperator & Some operator test scripts were named incorrectly > -- > > Key: AIRFLOW-3203 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3203 > Project: Apache Airflow > Issue Type: Bug > Components: operators, tests >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based > on documentation of Python package "docker". > In addition, its test is not really working due to incorrect file name. This > also happens for some other test scripts for Operators. This results in test > discovery failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3203) Bugs in DockerOperator & some operator tests
[ https://issues.apache.org/jira/browse/AIRFLOW-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3203: --- Description: Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based on documentation of Python package "docker". In addition, its test is not really working due to incorrect file name. This also happens for some other test scripts for Operators. was: Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based on documentation of Python package "docker". In addition, its test is not really working due to incorrect file name. Summary: Bugs in DockerOperator & some operator tests (was: Bugs in DockerOperator & its test) > Bugs in DockerOperator & some operator tests > > > Key: AIRFLOW-3203 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3203 > Project: Apache Airflow > Issue Type: Bug > Components: operators, tests >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based > on documentation of Python package "docker". > In addition, its test is not really working due to incorrect file name. This > also happens for some other test scripts for Operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3203) Bugs in DockerOperator & its test
[ https://issues.apache.org/jira/browse/AIRFLOW-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3203: --- Affects Version/s: 1.10.0 > Bugs in DockerOperator & its test > - > > Key: AIRFLOW-3203 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3203 > Project: Apache Airflow > Issue Type: Bug > Components: operators, tests >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based > on documentation of Python package "docker". > In addition, its test is not really working due to incorrect file name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3203) Bugs in DockerOperator & its test
Xiaodong DENG created AIRFLOW-3203: -- Summary: Bugs in DockerOperator & its test Key: AIRFLOW-3203 URL: https://issues.apache.org/jira/browse/AIRFLOW-3203 Project: Apache Airflow Issue Type: Bug Components: operators, tests Reporter: Xiaodong DENG Assignee: Xiaodong DENG Usage of `cpu_shares` and `cpu_shares` is incorrect in DockerOperator, based on documentation of Python package "docker". In addition, its test is not really working due to incorrect file name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3183) Potential Bug in utils/dag_processing/DagFileProcessorManager.max_runs_reached()
Xiaodong DENG created AIRFLOW-3183: -- Summary: Potential Bug in utils/dag_processing/DagFileProcessorManager.max_runs_reached() Key: AIRFLOW-3183 URL: https://issues.apache.org/jira/browse/AIRFLOW-3183 Project: Apache Airflow Issue Type: Bug Reporter: Xiaodong DENG Assignee: Xiaodong DENG In [https://github.com/apache/incubator-airflow/blob/df7c16a3ce01625277dd2e5c4ce4ed096dcfbb40/airflow/utils/dag_processing.py#L581,] the condition is to ensure the function will return False if any file's run_count is smaller than max_run. But the operator used here is "!=". Instead, it should be "<". This is because in *DagFileProcessorManager*, there is no statement helping limit the upper limit of run_count. It's possible that files' run_count will be bigger than max_run. In such case, max_runs_reached() method may fail its purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3175) Fix docstring format in airflow/jobs.py
Xiaodong DENG created AIRFLOW-3175: -- Summary: Fix docstring format in airflow/jobs.py Key: AIRFLOW-3175 URL: https://issues.apache.org/jira/browse/AIRFLOW-3175 Project: Apache Airflow Issue Type: Improvement Reporter: Xiaodong DENG Assignee: Xiaodong DENG Some minor issues in docstring format in airflow/jobs.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3174) Refine Docstring for SQL Operators
Xiaodong DENG created AIRFLOW-3174:
--
Summary: Refine Docstring for SQL Operators
Key: AIRFLOW-3174
URL: https://issues.apache.org/jira/browse/AIRFLOW-3174
Project: Apache Airflow
Issue Type: Improvement
Components: operators
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
SQL operators (based on {{DbApiHook}}) receive {{sql}} and {{parameters}}
(optional) as arguments. But {{parameters}} were not documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3148) Argument "parameters" in RedshiftToS3Transfer is not necessary
Xiaodong DENG created AIRFLOW-3148: -- Summary: Argument "parameters" in RedshiftToS3Transfer is not necessary Key: AIRFLOW-3148 URL: https://issues.apache.org/jira/browse/AIRFLOW-3148 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Argument "parameters" in RedshiftToS3Transfer is not necessary. Parameters are used to help render the SQL command. But in this operator, only schema and table are needed. There is no SQL command to render. Actually by checking the code, we can also realise argument "parameters" is never really used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any
[
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-3139:
---
Description:
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
SqliteOperator, PostgresOperator, the SQL command itself will be printed into
log INFO.
But if parameters are used for the sql command, the parameters are not printed.
This makes the logging less useful.
*Example:*
!Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
was:
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
SqliteOperator, PostgresOperator, the SQL command itself will be printed into
log INFO.
But if parameters are used for the sql command, the parameters are not printed.
This makes the printing useless.
*Example:*
!Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into
> log INFO.
> But if parameters are used for the sql command, the parameters are not
> printed. This makes the logging less useful.
>
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any
[
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-3139:
---
Description:
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
SqliteOperator, PostgresOperator, the SQL command itself will be printed into
log INFO.
But if parameters are used for the sql command, the parameters are not printed.
This makes the printing useless.
*Example:*
!Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
was:
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
SqliteOperator, PostgresOperator, the SQL command itself will be printed into
log INFO.
But if parameters are used for the sql command, the parameters are not printed.
This makes the printing useless.
*Example:*
!Screen Shot 2018-10-02 at 3.24.21 PM.png!
> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into
> log INFO.
> But if parameters are used for the sql command, the parameters are not
> printed. This makes the printing useless.
>
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any
Xiaodong DENG created AIRFLOW-3139:
--
Summary: Include parameters into log.info for SQL-operators if any
Key: AIRFLOW-3139
URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
Project: Apache Airflow
Issue Type: Improvement
Components: operators
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like
SqliteOperator, PostgresOperator, the SQL command itself will be printed into
log INFO.
But if parameters are used for the sql command, the parameters are not printed.
This makes the printing useless.
*Example:*
!Screen Shot 2018-10-02 at 3.24.21 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3127) Celery SSL Documentation is out-dated
Xiaodong DENG created AIRFLOW-3127: -- Summary: Celery SSL Documentation is out-dated Key: AIRFLOW-3127 URL: https://issues.apache.org/jira/browse/AIRFLOW-3127 Project: Apache Airflow Issue Type: Improvement Components: Documentation Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Now in `airflow.cfg`, for Celery-SSL, the items names are "ssl_active", "ssl_key", "ssl_cert", and "ssl_cacert". (since PR https://github.com/apache/incubator-airflow/pull/2806/files) But in the documentation [https://airflow.incubator.apache.org/security.html?highlight=celery] or [https://github.com/apache/incubator-airflow/blob/master/docs/security.rst,] it's "CELERY_SSL_ACTIVE", "CELERY_SSL_KEY", "CELERY_SSL_CERT", and "CELERY_SSL_CACERT", which is out-dated. This may confuse readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG reopened AIRFLOW-3104: > .airflowignore feature is not mentioned at all in documentation > --- > > Key: AIRFLOW-3104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 2.0.0 > > > This is a nice feature, but not mentioned at all in anywhere of the > documentation. > Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG resolved AIRFLOW-3067. Resolution: Fixed Fixed in commit https://github.com/apache/incubator-airflow/commit/19515e816785a6be828ec0191b9a7d83c9648110 > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-3067. -- Resolution: Fixed > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG reopened AIRFLOW-3067: > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Summary: (www_rbac) Flask flash messages are not displayed properly (no background color) (was: (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)) > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Description: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuilder would assign CSS class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). *Latest update:* we update bootstrap-theme.css, to add CSS classes 'alert-message' and 'alert-error'. This may help minimize the changes we need to do. was: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuilder would assign CSS class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625031#comment-16625031 ] Xiaodong DENG commented on AIRFLOW-3104: I have to close this ticket now by myself as I keep receiving email notifications due to the magical unicorn keeps changing assignee back and forth. It's spamming my mailbox. > .airflowignore feature is not mentioned at all in documentation > --- > > Key: AIRFLOW-3104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 2.0.0 > > > This is a nice feature, but not mentioned at all in anywhere of the > documentation. > Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-3104. -- Resolution: Fixed Fix Version/s: 2.0.0 > .airflowignore feature is not mentioned at all in documentation > --- > > Key: AIRFLOW-3104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 2.0.0 > > > This is a nice feature, but not mentioned at all in anywhere of the > documentation. > Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3104: --- Component/s: Documentation > .airflowignore feature is not mentioned at all in documentation > --- > > Key: AIRFLOW-3104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > This is a nice feature, but not mentioned at all in anywhere of the > documentation. > Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
Xiaodong DENG created AIRFLOW-3104: -- Summary: .airflowignore feature is not mentioned at all in documentation Key: AIRFLOW-3104 URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 Project: Apache Airflow Issue Type: Improvement Reporter: Xiaodong DENG Assignee: Xiaodong DENG This is a nice feature, but not mentioned at all in anywhere of the documentation. Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3048) (www_rbac) Non-Admin Roles' Access to Pages "Profile", "Edit User", and "Reset my password"
[
https://issues.apache.org/jira/browse/AIRFLOW-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624365#comment-16624365
]
Xiaodong DENG commented on AIRFLOW-3048:
I closed the PR I drafted earlier, given it can't ideally address this issue
(can refer to the GitHub PR page for detailed discussion).
But let's leave this Jira ticket open, in case anyone has any good idea later.
> (www_rbac) Non-Admin Roles' Access to Pages "Profile", "Edit User", and
> "Reset my password"
> ---
>
> Key: AIRFLOW-3048
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3048
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webapp
>Affects Versions: 1.10.0
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: attachment 1.png, attachment 2.png
>
>
> In the *www_rbac* app, at the top right corner, users can click to access
> "_*Profile*_" page, and in that page users are able to check their own
> profile, and further, they can change their own profile via _*Edit User*_
> (last and first name) and reset their OWN password ("*Reset my password*").
> However, in the current version (both 1.10 and master branch), the *default*
> role-view-permission only allows Admin roles to access this page.
> Of course the Admin user can add access to these pages for other roles, but
> it should not be the case actually (it should be there by default).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3048) (www_rbac) Non-Admin Roles' Access to Pages "Profile", "Edit User", and "Reset my password"
[
https://issues.apache.org/jira/browse/AIRFLOW-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG reassigned AIRFLOW-3048:
--
Assignee: (was: Xiaodong DENG)
> (www_rbac) Non-Admin Roles' Access to Pages "Profile", "Edit User", and
> "Reset my password"
> ---
>
> Key: AIRFLOW-3048
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3048
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webapp
>Affects Versions: 1.10.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Attachments: attachment 1.png, attachment 2.png
>
>
> In the *www_rbac* app, at the top right corner, users can click to access
> "_*Profile*_" page, and in that page users are able to check their own
> profile, and further, they can change their own profile via _*Edit User*_
> (last and first name) and reset their OWN password ("*Reset my password*").
> However, in the current version (both 1.10 and master branch), the *default*
> role-view-permission only allows Admin roles to access this page.
> Of course the Admin user can add access to these pages for other roles, but
> it should not be the case actually (it should be there by default).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params
[ https://issues.apache.org/jira/browse/AIRFLOW-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-2885. -- Resolution: Won't Fix It would not cause any real exception in the UI. Even if it's very 'implicit'. > A Bug in www_rbac.utils.get_params > -- > > Key: AIRFLOW-2885 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2885 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *get_params(page=0, search="abc",showPaused=False)* returns > "_search=abc=False_", while it's supposed to return > "page=0=abc=False". > This is because Python takes 0 as False when it's used in a conditional > statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params
[ https://issues.apache.org/jira/browse/AIRFLOW-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624363#comment-16624363 ] Xiaodong DENG commented on AIRFLOW-2885: Hi [~xnuinside] I realized this issue would not result in any actual exception in the UI (even it disobeys the zen "explicit is better than implicit" ;)) , so I closed the PR. Will close this Jira ticket now as well. Thanks for going through the tickets! > A Bug in www_rbac.utils.get_params > -- > > Key: AIRFLOW-2885 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2885 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *get_params(page=0, search="abc",showPaused=False)* returns > "_search=abc=False_", while it's supposed to return > "page=0=abc=False". > This is because Python takes 0 as False when it's used in a conditional > statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3073) A note is needed in 'Data Profiling' doc page to reminder users it's no longer supported in new webserver UI
[ https://issues.apache.org/jira/browse/AIRFLOW-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3073: --- Description: In [https://airflow.incubator.apache.org/profiling.html,] it's not mentioned at all that these features are no longer supported in new webser (FAB-based) due to security concern (https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#breaking-changes). (was: In [https://airflow.incubator.apache.org/profiling.html,] it's not mentioned at all that these features are no longer supported in new webser (FAB-based) due to security concern.) > A note is needed in 'Data Profiling' doc page to reminder users it's no > longer supported in new webserver UI > > > Key: AIRFLOW-3073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3073 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In [https://airflow.incubator.apache.org/profiling.html,] it's not mentioned > at all that these features are no longer supported in new webser (FAB-based) > due to security concern > (https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#breaking-changes). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3073) A note is needed in 'Data Profiling' doc page to reminder users it's no longer supported in new webserver UI
Xiaodong DENG created AIRFLOW-3073: -- Summary: A note is needed in 'Data Profiling' doc page to reminder users it's no longer supported in new webserver UI Key: AIRFLOW-3073 URL: https://issues.apache.org/jira/browse/AIRFLOW-3073 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Xiaodong DENG Assignee: Xiaodong DENG In [https://airflow.incubator.apache.org/profiling.html,] it's not mentioned at all that these features are no longer supported in new webser (FAB-based) due to security concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3070) Refine web UI authentication-related docs
Xiaodong DENG created AIRFLOW-3070:
--
Summary: Refine web UI authentication-related docs
Key: AIRFLOW-3070
URL: https://issues.apache.org/jira/browse/AIRFLOW-3070
Project: Apache Airflow
Issue Type: Improvement
Components: Documentation
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
Now in Airflow 1.10 we're already providing older version of web UI and new
FAB-based UI at the same time. But the documentation is not differentiated very
well. For example,
* this doc [https://airflow.apache.org/security.html#password] is only
applicable for old web UI only, but it's not hightlighted.
* command line tool {{create_user}} is only for new FAB-based UI only, it's
not highlighted as well.
This may be confusing to users, especially given not everyone is aware of the
existence of two UIs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Description: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuilder would assign CSS class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). was: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuild would assign class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). > (www_rbac) Flask flash messages are not categoried properly (result in > 'incomplete' UI display) > --- > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Description: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuild would assign class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). was: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. Flask AppBuild would assign class `alert-[category]` to that message when dispalying it (ref: https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html ). In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). > (www_rbac) Flask flash messages are not categoried properly (result in > 'incomplete' UI display) > --- > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuild would assign > class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Attachment: 2.png 1.png Summary: (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display) (was: (www_rbac) Flask flash messages not categoried (result in 'incomplete' UI display)) > (www_rbac) Flask flash messages are not categoried properly (result in > 'incomplete' UI display) > --- > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > This makes the the flash messages in the _*www_rbac*_ UI come with no > background color (please check the screenshots attached. We don't see this > issue in _*www*_ UI as all flash messages without categories are classed into > 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. Flask AppBuild would assign class > `alert-[category]` to that message when dispalying it (ref: > https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html > ). In bootstrap-theme.css, we don't have `alert-error` (we have > `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3067) (www_rbac) Flask flash messages not categoried (result in 'incomplete' UI display)
Xiaodong DENG created AIRFLOW-3067: -- Summary: (www_rbac) Flask flash messages not categoried (result in 'incomplete' UI display) Key: AIRFLOW-3067 URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 Project: Apache Airflow Issue Type: Improvement Components: webapp Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. Flask AppBuild would assign class `alert-[category]` to that message when dispalying it (ref: https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html ). In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3048) (www_rbac) Non-Admin Roles' Access to Pages "Profile", "Edit User", and "Reset my password"
Xiaodong DENG created AIRFLOW-3048:
--
Summary: (www_rbac) Non-Admin Roles' Access to Pages "Profile",
"Edit User", and "Reset my password"
Key: AIRFLOW-3048
URL: https://issues.apache.org/jira/browse/AIRFLOW-3048
Project: Apache Airflow
Issue Type: Improvement
Components: webapp
Affects Versions: 1.10.0
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
Attachments: attachment 1.png, attachment 2.png
In the *www_rbac* app, at the top right corner, users can click to access
"_*Profile*_" page, and in that page users are able to check their own profile,
and further, they can change their own profile via _*Edit User*_ (last and
first name) and reset their OWN password ("*Reset my password*").
However, in the current version (both 1.10 and master branch), the *default*
role-view-permission only allows Admin roles to access this page.
Of course the Admin user can add access to these pages for other roles, but it
should not be the case actually (it should be there by default).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2904) Clean an unnecessary line in airflow/executors/celery_executor.py
[ https://issues.apache.org/jira/browse/AIRFLOW-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2904: --- Fix Version/s: (was: 2.0.0) 1.10.1 > Clean an unnecessary line in airflow/executors/celery_executor.py > - > > Key: AIRFLOW-2904 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2904 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Trivial > Fix For: 1.10.1 > > Attachments: Screen Shot 2018-08-15 at 10.23.05 PM.png > > > Line > [https://github.com/apache/incubator-airflow/blob/00adade4163197fcff799168398edc629f989d34/airflow/executors/celery_executor.py#L34] > is unnecessary. Variable *PARALLELISM* is defined here but never used later. > !Screen Shot 2018-08-15 at 10.23.05 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2855) Need to Check Validity of Cron Expression When Process DAG File/Zip File
[ https://issues.apache.org/jira/browse/AIRFLOW-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2855: --- Fix Version/s: 1.10.1 > Need to Check Validity of Cron Expression When Process DAG File/Zip File > > > Key: AIRFLOW-2855 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2855 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 1.10.1 > > > *schedule_interval* of DAGs can either be *timedelta* or a *Cron expression*. > When it's a Cron expression, there is no mechanism to check its validity at > this moment. If there is anything wrong with the Cron expression itself, it > will cause issues when methods _*following_schedule(**)*_ and > _*previous_schedule()*_ are invoked (will affect scheduling). However, > exceptions will only be written into logs. From Web UI, it’s hard for users > to identify this issue & the source while no new task can be initiated > (especially for users who’re not very familiar with Cron). > It may be good to show error messages in web UI when a DAG's Cron expression > (as schedule_interval) can not be parsed by *croniter* properly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2825) S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3
[ https://issues.apache.org/jira/browse/AIRFLOW-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2825: --- Fix Version/s: (was: 2.0.0) 1.10.1 > S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase > ext in S3 > --- > > Key: AIRFLOW-2825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2825 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 1.10.1 > > > Because upper/lower case was not considered in the extension check, > S3ToHiveTransfer operator may think a GZIP file with uppercase ext `.GZ` is > not a GZIP file and raise exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2818) Cant' identify DAG .py files with capital case extension ('.PY', '.Py', '.pY')
[
https://issues.apache.org/jira/browse/AIRFLOW-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG closed AIRFLOW-2818.
--
Resolution: Won't Fix
> Cant' identify DAG .py files with capital case extension ('.PY', '.Py', '.pY')
> --
>
> Key: AIRFLOW-2818
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2818
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> This bug is quite simple.
> In multiple scripts in which we try to identify and list python files, the
> capital case extension was not considered. For example, if I have a DAG file
> with file name "dag.py", it will be identified. However, if I change the name
> into "dag.PY", it will be missed by Airflow.
> This is because simply using function *os.path.splitext()* can not handle
> case issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2922) Potential deal-lock bug in CeleryExecutor()
[ https://issues.apache.org/jira/browse/AIRFLOW-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2922: --- Fix Version/s: (was: 2.0.0) 1.10.1 > Potential deal-lock bug in CeleryExecutor() > --- > > Key: AIRFLOW-2922 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2922 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > Fix For: 1.10.1 > > > Code: > [https://github.com/apache/incubator-airflow/blob/404be4b021e803239f48e4dd5705759cc77aeb92/airflow/executors/celery_executor.py] > > Celery Task states normally change in ways like “PENDING -> STARTED -> > SUCCESS/FAILURE” > ([http://docs.celeryproject.org/en/latest/reference/celery.states.html]). > In lines 107 and 108, it’s `task.state` rather than `state`, i.e. it will > reflect the latest real-time state of the Celery task. > Let’s imagine: task state becomes “STARTED” (initial state is “PENDING”), > then the if-elif-else block will be triggered. It’s possible the Celery task > state becomes “SUCCESS” when whichever line between 94-105 is running. At > line 108, the latest state of the task in _*self.last_state*_ will be changed > to “SUCCESS” rather than “STARTED” because it’s referring to `task.state` > rather than variable `state`. > Then this task will be dead-locked because the if-elif-else block will never > be triggered for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2848) dag_id is missing in metadata table "job" for LocalTaskJob
[ https://issues.apache.org/jira/browse/AIRFLOW-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2848: --- Fix Version/s: (was: 2.0.0) 1.10.1 > dag_id is missing in metadata table "job" for LocalTaskJob > -- > > Key: AIRFLOW-2848 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2848 > Project: Apache Airflow > Issue Type: Bug > Components: db >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 1.10.1 > > Attachments: After this fix.png, Before this fix.png > > > dag_id is missing for all entries in metadata table "job" with job_type > "LocalTaskJob". > This is due to that dag_id was not specified within class LocalTaskJob. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2999) S3_hook - add the ability to download file to local disk
[ https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605185#comment-16605185 ] Xiaodong DENG commented on AIRFLOW-2999: Hi [~jackjack10], maybe I'll leave this to one of the committers to decide if we want to go for this? I would be more than happy to take this up if the committers agree with this. > S3_hook - add the ability to download file to local disk > - > > Key: AIRFLOW-2999 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2999 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > > The [S3_hook > |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177] > has get_key method that returns boto3.s3.Object it also has load_file method > which loads file from local file system to S3. > > What it doesn't have is a method to download a file from S3 to the local file > system. > Basicly it should be something very simple... an extention to the get_key > method with parameter to the destination on local file system adding a code > for taking the boto3.s3.Object and save it on the disk. Note: that it can be > more than 1 file if the user choose a folder in S3. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2999) S3_hook - add the ability to download file to local disk
[
https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603805#comment-16603805
]
Xiaodong DENG commented on AIRFLOW-2999:
Hi [~jackjack10] , thanks for tagging me. I guess we don't need a separate
method for this.
There is an existing method in S3Hook(),
[read_key()|[https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L193],]
which should suffice for your use case.
Say you have object "test_path/test.txt" in S3 to download, you only need to
run something like this
{code:java}
hook = S3Hook(...)
with open("localfile_name_you_want.txt", "w") as f:
f.write(hook.read_key(key="s3://your_bucket_name/test_path/test.txt"))
{code}
This is also quite consistent with how we normally write file into local file
system in Python.
Please let me know whether this makes sense to you. Thanks
[~ashb] any other thoghts?
> S3_hook - add the ability to download file to local disk
> -
>
> Key: AIRFLOW-2999
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2999
> Project: Apache Airflow
> Issue Type: Task
>Affects Versions: 1.10.0
>Reporter: jack
>Priority: Major
>
> The [S3_hook
> |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177]
> has get_key method that returns boto3.s3.Object it also has load_file method
> which loads file from local file system to S3.
>
> What it doesn't have is a method to download a file from S3 to the local file
> system.
> Basicly it should be something very simple... an extention to the get_key
> method with parameter to the destination on local file system adding a code
> for taking the boto3.s3.Object and save it on the disk. Note: that it can be
> more than 1 file if the user choose a folder in S3.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2985) Operators for S3 object copying/deleting [boto3.client.copy_object()/delete_object()]
[ https://issues.apache.org/jira/browse/AIRFLOW-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2985: --- Description: Currently we don't have an operator in Airflow to help copy/delete objects within S3, while they may be quite common use case when we deal with the data in S3. (was: Currently we don't have an operator in Airflow to help copy objects within S3, while this is a quite common use case when we deal with the data in S3.) > Operators for S3 object copying/deleting > [boto3.client.copy_object()/delete_object()] > - > > Key: AIRFLOW-2985 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2985 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > Currently we don't have an operator in Airflow to help copy/delete objects > within S3, while they may be quite common use case when we deal with the data > in S3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2965) Add CLI command to find the next dag run.
[
https://issues.apache.org/jira/browse/AIRFLOW-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599661#comment-16599661
]
Xiaodong DENG commented on AIRFLOW-2965:
Hi [~sanand] and [~jackjack10], I would like to take this up. Will try to draft
a PR later.
> Add CLI command to find the next dag run.
> -
>
> Key: AIRFLOW-2965
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2965
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.1
>
>
> I have a dag with the following properties:
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 1 * * *',
> max_active_runs=1,
> catchup=False){code}
>
>
> This runs great.
> Last run is: 2018-08-26 01:00 (start date is 2018-08-27 01:00)
>
> Now it's 2018-08-27 17:55 I decided to change my dag to:
>
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 23 * * *',
> max_active_runs=1,
> catchup=False){code}
>
> Now, I have no idea when will be the next dag run.
> Will it be today at 23:00? I can't be sure when the cycle is complete. I'm
> not even sure that this change will do what I wish.
> I'm sure you guys are expert and you can answer this question but most of us
> wouldn't know.
>
> The scheduler has the knowledge when the dag is available for running. All
> I'm asking is to take that knowledge and create a CLI command that I will
> give the dag_id and it will tell me the next date/hour which my dag will be
> runnable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2965) Add CLI command to find the next dag run.
[
https://issues.apache.org/jira/browse/AIRFLOW-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG reassigned AIRFLOW-2965:
--
Assignee: Xiaodong DENG
> Add CLI command to find the next dag run.
> -
>
> Key: AIRFLOW-2965
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2965
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.1
>
>
> I have a dag with the following properties:
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 1 * * *',
> max_active_runs=1,
> catchup=False){code}
>
>
> This runs great.
> Last run is: 2018-08-26 01:00 (start date is 2018-08-27 01:00)
>
> Now it's 2018-08-27 17:55 I decided to change my dag to:
>
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 23 * * *',
> max_active_runs=1,
> catchup=False){code}
>
> Now, I have no idea when will be the next dag run.
> Will it be today at 23:00? I can't be sure when the cycle is complete. I'm
> not even sure that this change will do what I wish.
> I'm sure you guys are expert and you can answer this question but most of us
> wouldn't know.
>
> The scheduler has the knowledge when the dag is available for running. All
> I'm asking is to take that knowledge and create a CLI command that I will
> give the dag_id and it will tell me the next date/hour which my dag will be
> runnable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2985) An operator for S3 object copying [boto3.client.copy_object()]
Xiaodong DENG created AIRFLOW-2985: -- Summary: An operator for S3 object copying [boto3.client.copy_object()] Key: AIRFLOW-2985 URL: https://issues.apache.org/jira/browse/AIRFLOW-2985 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG Currently we don't have an operator in Airflow to help copy objects within S3, while this is a quite common use case when we deal with the data in S3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2959) HTTPSensor doc/comment is not clear enough
[ https://issues.apache.org/jira/browse/AIRFLOW-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2959: --- Description: The HTTPSensor is supposed to sense if a resource/endpoint if available at a location specified by http connection. It will return False when the HTTP Status code is 404 or check function returns False. However, HTTP Error code other than 404, or Connection Refused, would fail the sensor itself directly (no more poking) [https://github.com/apache/incubator-airflow/blob/f6191fbd7a61766bd002873db371b74702b13ff3/airflow/sensors/http_sensor.py#L90]. This is not clear enough in the doc. Users like myself may think these situations will also return False and try poking later, while it's not true. This should be made clear in the documetation. was: The HTTPSensor is supposed to sense if a resource/endpoint if available at a location specified by http connection. It will return False when the HTTP Status code is 404 or check function returns False. However, HTTP Error code other than 404, or Connection Refused, would fail the sensor itself directly (no more poking). This is not clear enough in the doc. Users like myself may think these situations will also return False and try poking later, while it's not true. This should be made clear in the documetation. > HTTPSensor doc/comment is not clear enough > -- > > Key: AIRFLOW-2959 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2959 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The HTTPSensor is supposed to sense if a resource/endpoint if available at a > location specified by http connection. It will return False when the HTTP > Status code is 404 or check function returns False. > However, HTTP Error code other than 404, or Connection Refused, would fail > the sensor itself directly (no more poking) > [https://github.com/apache/incubator-airflow/blob/f6191fbd7a61766bd002873db371b74702b13ff3/airflow/sensors/http_sensor.py#L90]. > This is not clear enough in the doc. Users like myself may think these > situations will also return False and try poking later, while it's not true. > This should be made clear in the documetation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2959) HTTPSensor doc/comment is not clear enough
Xiaodong DENG created AIRFLOW-2959: -- Summary: HTTPSensor doc/comment is not clear enough Key: AIRFLOW-2959 URL: https://issues.apache.org/jira/browse/AIRFLOW-2959 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG The HTTPSensor is supposed to sense if a resource/endpoint if available at a location specified by http connection. It will return False when the HTTP Status code is 404 or check function returns False. However, HTTP Error code other than 404, or Connection Refused, would fail the sensor itself directly (no more poking). This is not clear enough in the doc. Users like myself may think these situations will also return False and try poking later, while it's not true. This should be made clear in the documetation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2948) Arg checking & better doc for SSHOperator and SFTPOperator
Xiaodong DENG created AIRFLOW-2948: -- Summary: Arg checking & better doc for SSHOperator and SFTPOperator Key: AIRFLOW-2948 URL: https://issues.apache.org/jira/browse/AIRFLOW-2948 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG There may be different combinations of arguments, and some processings are being done 'silently', while users may not be fully aware of them. For example - User only needs to provide either `ssh_hook` or `ssh_conn_id`, while this is not clear in doc - if both provided, `ssh_conn_id` will be ignored. - if `remote_host` is provided, it will replace the `remote_host` which wasndefined in `ssh_hook` or predefined in the connection of `ssh_conn_id` These should be documented clearly to ensure it's transparent to the users. log.info() should also be used to remind users and provide clear logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2922) Potential deal-lock bug in CeleryExecutor()
Xiaodong DENG created AIRFLOW-2922: -- Summary: Potential deal-lock bug in CeleryExecutor() Key: AIRFLOW-2922 URL: https://issues.apache.org/jira/browse/AIRFLOW-2922 Project: Apache Airflow Issue Type: Improvement Components: executor Reporter: Xiaodong DENG Assignee: Xiaodong DENG Code: [https://github.com/apache/incubator-airflow/blob/404be4b021e803239f48e4dd5705759cc77aeb92/airflow/executors/celery_executor.py] Celery Task states normally change in ways like “PENDING -> STARTED -> SUCCESS/FAILURE” ([http://docs.celeryproject.org/en/latest/reference/celery.states.html]). In lines 107 and 108, it’s `task.state` rather than `state`, i.e. it will reflect the latest real-time state of the Celery task. Let’s imagine: task state becomes “STARTED” (initial state is “PENDING”), then the if-elif-else block will be triggered. It’s possible the Celery task state becomes “SUCCESS” when whichever line between 94-105 is running. At line 108, the latest state of the task in _*self.last_state*_ will be changed to “SUCCESS” rather than “STARTED” because it’s referring to `task.state` rather than variable `state`. Then this task will be dead-locked because the if-elif-else block will never be triggered for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2921) A trivial incorrectness in CeleryExecutor()
[ https://issues.apache.org/jira/browse/AIRFLOW-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2921: --- Summary: A trivial incorrectness in CeleryExecutor() (was: A minor incorrectness in CeleryExecutor()) > A trivial incorrectness in CeleryExecutor() > --- > > Key: AIRFLOW-2921 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2921 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > Code: > [https://github.com/apache/incubator-airflow/blob/62bd68999bade4261daf0499b56098d2e6fc9470/airflow/executors/celery_executor.py#L108] > The logic here is: if a task state becomes either SUCCESS or FAILURE or > REVOKED, it will be removed from *self.tasks()* and *self.last_state()*. > However, because line108 is not indented properly, this task will be added > back to *self.last_state()* again. > This will not lead to any error. But may still be worth changing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2921) A trivial incorrectness in CeleryExecutor()
[ https://issues.apache.org/jira/browse/AIRFLOW-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2921: --- Priority: Trivial (was: Minor) > A trivial incorrectness in CeleryExecutor() > --- > > Key: AIRFLOW-2921 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2921 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Trivial > > Code: > [https://github.com/apache/incubator-airflow/blob/62bd68999bade4261daf0499b56098d2e6fc9470/airflow/executors/celery_executor.py#L108] > The logic here is: if a task state becomes either SUCCESS or FAILURE or > REVOKED, it will be removed from *self.tasks()* and *self.last_state()*. > However, because line108 is not indented properly, this task will be added > back to *self.last_state()* again. > This will not lead to any error. But may still be worth changing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2921) A minor incorrectness in CeleryExecutor()
Xiaodong DENG created AIRFLOW-2921: -- Summary: A minor incorrectness in CeleryExecutor() Key: AIRFLOW-2921 URL: https://issues.apache.org/jira/browse/AIRFLOW-2921 Project: Apache Airflow Issue Type: Improvement Reporter: Xiaodong DENG Assignee: Xiaodong DENG Code: [https://github.com/apache/incubator-airflow/blob/62bd68999bade4261daf0499b56098d2e6fc9470/airflow/executors/celery_executor.py#L108] The logic here is: if a task state becomes either SUCCESS or FAILURE or REVOKED, it will be removed from *self.tasks()* and *self.last_state()*. However, because line108 is not indented properly, this task will be added back to *self.last_state()* again. This will not lead to any error. But may still be worth changing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2916) Add argument `verify` for AwsHook() and S3 related sensors/operators
Xiaodong DENG created AIRFLOW-2916: -- Summary: Add argument `verify` for AwsHook() and S3 related sensors/operators Key: AIRFLOW-2916 URL: https://issues.apache.org/jira/browse/AIRFLOW-2916 Project: Apache Airflow Issue Type: Improvement Components: hooks, operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG The AwsHook() and S3-related operators/sensors are depending on package boto3. In boto3, when we initiate a client or a resource, argument `verify` is provided (https://boto3.readthedocs.io/en/latest/reference/core/session.html ). It is useful when # users want to use a different CA cert bundle than the one used by botocore. # users want to have '--no-verify-ssl'. This is especially useful when we're using on-premises S3 or other implementations of object storage, like IBM's Cloud Object Storage. However, this feature is not provided in Airflow for S3 yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2913) Check bucket_key and bucket_name combination in S3KeySensor()
Xiaodong DENG created AIRFLOW-2913: -- Summary: Check bucket_key and bucket_name combination in S3KeySensor() Key: AIRFLOW-2913 URL: https://issues.apache.org/jira/browse/AIRFLOW-2913 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG 1. When *bucket_name* is provided, and *bucket_key* is also provided as a full S3:// url, the full_url obtained eventually will be wrong. It will be like "_s3://bucket_name/s3://bucket_name/object_key_". This should be avoided by adding checking and raise exception in such case. 2. Given the documentation is not clear enough, this may happen to new users of S3KeySensor() (like myself). I have improved the Documentation as well. 3. Unit tests are added for the bucket_key and bucket_name combination checking as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2904) Clean an unnecessary line in airflow/executors/celery_executor.py
Xiaodong DENG created AIRFLOW-2904: -- Summary: Clean an unnecessary line in airflow/executors/celery_executor.py Key: AIRFLOW-2904 URL: https://issues.apache.org/jira/browse/AIRFLOW-2904 Project: Apache Airflow Issue Type: Improvement Components: executor Reporter: Xiaodong DENG Assignee: Xiaodong DENG Attachments: Screen Shot 2018-08-15 at 10.23.05 PM.png Line [https://github.com/apache/incubator-airflow/blob/00adade4163197fcff799168398edc629f989d34/airflow/executors/celery_executor.py#L34] is unnecessary. Variable *PARALLELISM* is defined here but never used later. !Screen Shot 2018-08-15 at 10.23.05 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2808) Plugin duplication checking is not working
[
https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG closed AIRFLOW-2808.
--
Resolution: Invalid
Can't pass Kubernetes tests properly (even though seems not relating to this
commit). Will need to check
> Plugin duplication checking is not working
> --
>
> Key: AIRFLOW-2808
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
> Project: Apache Airflow
> Issue Type: Bug
> Components: plugins
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> h2. *Background*
> A plugin duplication checking was designed in *plugins_manager.py*
> [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
> .
> Corresponding commit was
> [https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
>
> However, it turns out that this checking is not really working (reason:
> plugin method object name is formed using plugin file path + plugin file name
> + Plugin Class name. It will never be duplicated given there will not be two
> files with the same name in the same directory).
> h2. *Issue*
> In my production environment, there are two plugin files with the same name
> and operator names in the new _AirflowPlugin_ classes defined inside.
> However, they passed the check without any warning or exception.
> For example, I have a plugin *file_sensor_1.py* as below,
> {code:java}
> from airflow.plugins_manager import AirflowPlugin
> from airflow.operators.sensors import BaseSensorOperator
> from airflow.utils.decorators import apply_defaults
> import os
> class local_file_sensor(BaseSensorOperator):
> @apply_defaults
> def __init__(self, file_path, *args, **kwargs):
> super(local_file_sensor, self).__init__(*args, **kwargs)
> self.file_path = file_path
> def poke(self, context):
> self.log.info('A-Poking: %s', self.file_path)
> return os.path.exists(self.file_path)
> class AirflowLocalFileSensorPlugin(AirflowPlugin):
> name = "local_file_sensor_plugin"
> operators = [local_file_sensor]
> {code}
>
> I copy & paste it into another plugin file *file_sensor_2.py*, and make the
> only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
> me check which one is picked).
> Only one plugin would be loaded eventually (because the earlier loaded one
> will be overwritten by the later loaded one
> [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
> ). However, which one? We don't know. It's indeterminate. So far the file
> name seems to be the only factor affecting which one would be picked by
> Airflow.
> h2. *My proposal*
> Give WARNING to the users when they launch the Airflow. (Or should we give
> error msg and fail the launching?)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2872) Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView()
[ https://issues.apache.org/jira/browse/AIRFLOW-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-2872. -- Resolution: Won't Fix > Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView() > -- > > Key: AIRFLOW-2872 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2872 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > To implement "Ad Hoc Query" in for RBAC in /www_rbac, based on the existing > implementation in /www. > In addition, refine the existing QueryView(): > # The ".csv" button in *Ad Hoc Query* view is responding with a plain text > file, rather than a CSV file (even though users can manually change the > extension). > # Argument 'has_data' passed to the template is not used by the template > 'airflow/query.html'. > # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced > before assignment' > # 'result = df.to_html()' should only be invoked when user doesn NOT choose > '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the > result it returns will not be used if user askes for CSV downloading instead > of a html page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2886) Secure Flask SECRET_KEY
[
https://issues.apache.org/jira/browse/AIRFLOW-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG closed AIRFLOW-2886.
--
Resolution: Fixed
Fixed with commit
https://github.com/apache/incubator-airflow/commit/f7602f8266559e55bc602a9639e3e1ab640f30e8
> Secure Flask SECRET_KEY
> ---
>
> Key: AIRFLOW-2886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2886
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651]
> and [https://github.com/apache/incubator-airflow/pull/3729] , I proposed to
> generate random SECRET_KEY for Flask App.
> If we have multiple workers for the Flask webserver, we may encounter CSRF
> error {{The CSRF session token is missing}} .
> On the other hand, it's still very important to have as random SECRET_KEY as
> possible for security reasons. We can deal with it like how we dealt with
> FERNET_KEY (i.e. generate a random value when the airflow.cfg file is
> initiated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2896) Improve HdfsSensor()
[ https://issues.apache.org/jira/browse/AIRFLOW-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-2896. -- Resolution: Invalid Hdfs is being refactored in [#3560|https://github.com/apache/incubator-airflow/pull/3560] and this is then already outdated > Improve HdfsSensor() > > > Key: AIRFLOW-2896 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2896 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > # Make documentation clearer (format for `ignored_ext` should be extensions > like ''py rather than '.py') > # Ensure upper/lower case would not affect the usage of `ignored_ext` > feagure. > # Add tests for methods filter_for_ignored_ext() and filter_for_filesize() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2896) Improve HdfsSensor()
Xiaodong DENG created AIRFLOW-2896: -- Summary: Improve HdfsSensor() Key: AIRFLOW-2896 URL: https://issues.apache.org/jira/browse/AIRFLOW-2896 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG # Make documentation clearer (format for `ignored_ext` should be extensions like ''py rather than '.py') # Ensure upper/lower case would not affect the usage of `ignored_ext` feagure. # Add tests for methods filter_for_ignored_ext() and filter_for_filesize() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2886) Secure Flask SECRET_KEY
Xiaodong DENG created AIRFLOW-2886:
--
Summary: Secure Flask SECRET_KEY
Key: AIRFLOW-2886
URL: https://issues.apache.org/jira/browse/AIRFLOW-2886
Project: Apache Airflow
Issue Type: Bug
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651] and
[https://github.com/apache/incubator-airflow/pull/3729] , I proposed to
generate random SECRET_KEY for Flask App.
If we have multiple workers for the Flask webserver, we may encounter CSRF
error {{The CSRF session token is missing}} .
On the other hand, it's still very important to have as random SECRET_KEY as
possible for security reasons. We can deal with it like how we dealt with
FERNET_KEY (i.e. generate a random value when the airflow.cfg file is
initiated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params
Xiaodong DENG created AIRFLOW-2885: -- Summary: A Bug in www_rbac.utils.get_params Key: AIRFLOW-2885 URL: https://issues.apache.org/jira/browse/AIRFLOW-2885 Project: Apache Airflow Issue Type: Bug Components: webserver Reporter: Xiaodong DENG Assignee: Xiaodong DENG *get_params(page=0, search="abc",showPaused=False)* returns "_search=abc=False_", while it's supposed to return "page=0=abc=False". This is because Python takes 0 as False when it's used in a conditional statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2884) Fix Flask SECRET_KEY security issue in www_rbac
Xiaodong DENG created AIRFLOW-2884: -- Summary: Fix Flask SECRET_KEY security issue in www_rbac Key: AIRFLOW-2884 URL: https://issues.apache.org/jira/browse/AIRFLOW-2884 Project: Apache Airflow Issue Type: Bug Components: webapp, webserver Reporter: Xiaodong DENG Assignee: Xiaodong DENG Flask secret key should be as random as possible, while it's not in Airflow Flask App. This issue was fixed for *www* in ticket https://issues.apache.org/jira/browse/AIRFLOW-2809 (merged in PR [https://github.com/apache/incubator-airflow/pull/3651)] . But this issue was not fixed for *www_rbac* yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
Xiaodong DENG created AIRFLOW-2878:
--
Summary: "/www_rbac" Layout/UI Display Issue
Key: AIRFLOW-2878
URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
Project: Apache Airflow
Issue Type: Improvement
Components: ui
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
The new RBAC UI has some issues about layout/UI display.
The header () is not shown ("hidden" by the Nav Bar), or tables are not
shown completely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2872) Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView()
[ https://issues.apache.org/jira/browse/AIRFLOW-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2872: --- Description: To implement "Ad Hoc Query" in for RBAC in /www_rbac, based on the existing implementation in /www. In addition, refine the existing QueryView(): # The ".csv" button in *Ad Hoc Query* view is responding with a plain text file, rather than a CSV file (even though users can manually change the extension). # Argument 'has_data' passed to the template is not used by the template 'airflow/query.html'. # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced before assignment' # 'result = df.to_html()' should only be invoked when user doesn NOT choose '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the result it returns will not be used if user askes for CSV downloading instead of a html page. was: # The ".csv" button in *Ad Hoc Query* view is responding with a plain text file, rather than a CSV file (even though users can manually change the extension). # Argument 'has_data' passed to the template is not used by the template 'airflow/query.html'. # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced before assignment' # 'result = df.to_html()' should only be invoked when user doesn NOT choose '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the result it returns will not be used if user askes for CSV downloading instead of a html page. Summary: Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView() (was: Minor bugs in "Ad Hoc Query" view, and refinement) > Implement "Ad Hoc Query" in /www_rbac, and refine existing QueryView() > -- > > Key: AIRFLOW-2872 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2872 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > To implement "Ad Hoc Query" in for RBAC in /www_rbac, based on the existing > implementation in /www. > In addition, refine the existing QueryView(): > # The ".csv" button in *Ad Hoc Query* view is responding with a plain text > file, rather than a CSV file (even though users can manually change the > extension). > # Argument 'has_data' passed to the template is not used by the template > 'airflow/query.html'. > # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced > before assignment' > # 'result = df.to_html()' should only be invoked when user doesn NOT choose > '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the > result it returns will not be used if user askes for CSV downloading instead > of a html page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2872) Minor bugs in "Ad Hoc Query" view, and refinement
Xiaodong DENG created AIRFLOW-2872: -- Summary: Minor bugs in "Ad Hoc Query" view, and refinement Key: AIRFLOW-2872 URL: https://issues.apache.org/jira/browse/AIRFLOW-2872 Project: Apache Airflow Issue Type: Improvement Components: ui Reporter: Xiaodong DENG Assignee: Xiaodong DENG # The ".csv" button in *Ad Hoc Query* view is responding with a plain text file, rather than a CSV file (even though users can manually change the extension). # Argument 'has_data' passed to the template is not used by the template 'airflow/query.html'. # Sometimes get errors 'UnboundLocalError: local variable 'df' referenced before assignment' # 'result = df.to_html()' should only be invoked when user doesn NOT choose '.csv'. Otherwise it's a waste of resource to invoke 'df.to_html()' since the result it returns will not be used if user askes for CSV downloading instead of a html page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2836) Minor improvement of contrib.sensors.FileSensor
[
https://issues.apache.org/jira/browse/AIRFLOW-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG resolved AIRFLOW-2836.
Resolution: Fixed
> Minor improvement of contrib.sensors.FileSensor
> ---
>
> Key: AIRFLOW-2836
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2836
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> h4. *Background*
> The default *fs_conn_id* in contrib.sensors.FileSensor is '_*fs_default2*_'.
> However, when we initiate the database
> (https://github.com/apache/incubator-airflow/blob/master/airflow/utils/db.py#L88),
> there isn't such an entry. It doesn't exist anywhere else.
> h4. *Issue*
> The purpose of _contrib.sensors.FileSensor_ is mainly for checking local file
> system (of course can also be used for NAS). Then the path ("/") from default
> connection 'fs_default' would suffice.
> However, given the default value for *fs_conn_id* in
> contrib.sensors.FileSensor is "fs_default2" (a value doesn't exist), it will
> make the situation much more complex.
> When users intend to check local file system only, they should be able to
> leave *fs_conn_id* default directly, instead of going setting up another
> connection separately.
> h4. Proposal
> Change default value for *fs_conn_id* in contrib.sensors.FileSensor from
> "fs_default2" to "fs_default" (actually in the related test, the *fs_conn_id*
> are all specified to be "fs_default").
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2855) Need to Check Validity of Cron Expression When Process DAG File/Zip File
[ https://issues.apache.org/jira/browse/AIRFLOW-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG resolved AIRFLOW-2855. Resolution: Fixed > Need to Check Validity of Cron Expression When Process DAG File/Zip File > > > Key: AIRFLOW-2855 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2855 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *schedule_interval* of DAGs can either be *timedelta* or a *Cron expression*. > When it's a Cron expression, there is no mechanism to check its validity at > this moment. If there is anything wrong with the Cron expression itself, it > will cause issues when methods _*following_schedule(**)*_ and > _*previous_schedule()*_ are invoked (will affect scheduling). However, > exceptions will only be written into logs. From Web UI, it’s hard for users > to identify this issue & the source while no new task can be initiated > (especially for users who’re not very familiar with Cron). > It may be good to show error messages in web UI when a DAG's Cron expression > (as schedule_interval) can not be parsed by *croniter* properly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2848) dag_id is missing in metadata table "job" for LocalTaskJob
[ https://issues.apache.org/jira/browse/AIRFLOW-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572561#comment-16572561 ] Xiaodong DENG commented on AIRFLOW-2848: Thanks [~xnuinside] for reminding. Have closed this ticket. > dag_id is missing in metadata table "job" for LocalTaskJob > -- > > Key: AIRFLOW-2848 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2848 > Project: Apache Airflow > Issue Type: Bug > Components: db >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Attachments: After this fix.png, Before this fix.png > > > dag_id is missing for all entries in metadata table "job" with job_type > "LocalTaskJob". > This is due to that dag_id was not specified within class LocalTaskJob. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2848) dag_id is missing in metadata table "job" for LocalTaskJob
[ https://issues.apache.org/jira/browse/AIRFLOW-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-2848. -- Resolution: Fixed > dag_id is missing in metadata table "job" for LocalTaskJob > -- > > Key: AIRFLOW-2848 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2848 > Project: Apache Airflow > Issue Type: Bug > Components: db >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Attachments: After this fix.png, Before this fix.png > > > dag_id is missing for all entries in metadata table "job" with job_type > "LocalTaskJob". > This is due to that dag_id was not specified within class LocalTaskJob. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2855) Need to Check Validity of Cron Expression When Process DAG File/Zip File
[ https://issues.apache.org/jira/browse/AIRFLOW-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2855: --- Description: *schedule_interval* of DAGs can either be *timedelta* or a *Cron expression*. When it's a Cron expression, there is no mechanism to check its validity at this moment. If there is anything wrong with the Cron expression itself, it will cause issues when methods _*following_schedule(**)*_ and _*previous_schedule()*_ are invoked (will affect scheduling). However, exceptions will only be written into logs. From Web UI, it’s hard for users to identify this issue & the source while no new task can be initiated (especially for users who’re not very familiar with Cron). It may be good to show error messages in web UI when a DAG's Cron expression (as schedule_interval) can not be parsed by *croniter* properly. was: *schedule_interval* of DAGs can either be *datetime* or a *Cron expression*. When it's a Cron expression, there is no mechanism to check its validity at this moment. If there is anything wrong with the Cron expression itself, it will cause issues when methods _*following_schedule(**)*_ and _*previous_schedule()*_ are invoked (will affect scheduling). However, exceptions will only be written into logs. From Web UI, it’s hard for users to identify this issue & the source while no new task can be initiated (especially for users who’re not very familiar with Cron). It may be good to show error messages in web UI when a DAG's Cron expression (as schedule_interval) can not be parsed by *croniter* properly. > Need to Check Validity of Cron Expression When Process DAG File/Zip File > > > Key: AIRFLOW-2855 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2855 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *schedule_interval* of DAGs can either be *timedelta* or a *Cron expression*. > When it's a Cron expression, there is no mechanism to check its validity at > this moment. If there is anything wrong with the Cron expression itself, it > will cause issues when methods _*following_schedule(**)*_ and > _*previous_schedule()*_ are invoked (will affect scheduling). However, > exceptions will only be written into logs. From Web UI, it’s hard for users > to identify this issue & the source while no new task can be initiated > (especially for users who’re not very familiar with Cron). > It may be good to show error messages in web UI when a DAG's Cron expression > (as schedule_interval) can not be parsed by *croniter* properly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2855) Need to Check Validity of Cron Expression When Process DAG File/Zip File
Xiaodong DENG created AIRFLOW-2855: -- Summary: Need to Check Validity of Cron Expression When Process DAG File/Zip File Key: AIRFLOW-2855 URL: https://issues.apache.org/jira/browse/AIRFLOW-2855 Project: Apache Airflow Issue Type: Improvement Components: DAG Reporter: Xiaodong DENG Assignee: Xiaodong DENG *schedule_interval* of DAGs can either be *datetime* or a *Cron expression*. When it's a Cron expression, there is no mechanism to check its validity at this moment. If there is anything wrong with the Cron expression itself, it will cause issues when methods _*following_schedule(**)*_ and _*previous_schedule()*_ are invoked (will affect scheduling). However, exceptions will only be written into logs. From Web UI, it’s hard for users to identify this issue & the source while no new task can be initiated (especially for users who’re not very familiar with Cron). It may be good to show error messages in web UI when a DAG's Cron expression (as schedule_interval) can not be parsed by *croniter* properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2848) dag_id is missing in metadata table "job" for LocalTaskJob
Xiaodong DENG created AIRFLOW-2848: -- Summary: dag_id is missing in metadata table "job" for LocalTaskJob Key: AIRFLOW-2848 URL: https://issues.apache.org/jira/browse/AIRFLOW-2848 Project: Apache Airflow Issue Type: Bug Components: db Reporter: Xiaodong DENG Assignee: Xiaodong DENG Attachments: After this fix.png, Before this fix.png dag_id is missing for all entries in metadata table "job" with job_type "LocalTaskJob". This is due to that dag_id was not specified within class LocalTaskJob. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2843) ExternalTaskSensor: Add option to cease waiting immediately if the external task doesn't exist
Xiaodong DENG created AIRFLOW-2843: -- Summary: ExternalTaskSensor: Add option to cease waiting immediately if the external task doesn't exist Key: AIRFLOW-2843 URL: https://issues.apache.org/jira/browse/AIRFLOW-2843 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG h2. Background *ExternalTaskSensor* will keep waiting (given restrictions of retries, poke_interval, etc), even if the external task specified doesn't exist at all. In some cases, this waiting may still make sense as new DAG may backfill. But it may be good to provide an option to cease waiting immediately if the external task specified doesn't exist. h2. Proposal Provide an argument "check_existence". Set to *True* to check if the external task exists, and immediately cease waiting if the external task does not exist. The default value is set to *False* (no check or ceasing will happen) so it will not affect any existing DAGs or user expectation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
