[jira] [Created] (AIRFLOW-7061) Rename openfass to openfaas
Bas Harenslak created AIRFLOW-7061: -- Summary: Rename openfass to openfaas Key: AIRFLOW-7061 URL: https://issues.apache.org/jira/browse/AIRFLOW-7061 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: 2.0.0, 1.10.10 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6962) Fix "compeleted" to "completed"
Bas Harenslak created AIRFLOW-6962: -- Summary: Fix "compeleted" to "completed" Key: AIRFLOW-6962 URL: https://issues.apache.org/jira/browse/AIRFLOW-6962 Project: Apache Airflow Issue Type: Bug Components: aws Affects Versions: 2.0.0, 1.10.10 Reporter: Bas Harenslak Assignee: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6886) Cleanup the SageMakerTrainingOperator result
[ https://issues.apache.org/jira/browse/AIRFLOW-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-6886: --- Issue Type: Improvement (was: Bug) > Cleanup the SageMakerTrainingOperator result > > > Key: AIRFLOW-6886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6886 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6886) Cleanup the SageMakerTrainingOperator result
[ https://issues.apache.org/jira/browse/AIRFLOW-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-6886: -- Assignee: Bas Harenslak > Cleanup the SageMakerTrainingOperator result > > > Key: AIRFLOW-6886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6886 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6886) Cleanup the SageMakerTrainingOperator result
Bas Harenslak created AIRFLOW-6886: -- Summary: Cleanup the SageMakerTrainingOperator result Key: AIRFLOW-6886 URL: https://issues.apache.org/jira/browse/AIRFLOW-6886 Project: Apache Airflow Issue Type: Bug Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6884) Make SageMakerTrainingOperator idempotent
Bas Harenslak created AIRFLOW-6884: -- Summary: Make SageMakerTrainingOperator idempotent Key: AIRFLOW-6884 URL: https://issues.apache.org/jira/browse/AIRFLOW-6884 Project: Apache Airflow Issue Type: Bug Components: aws Affects Versions: 2.0.0, 1.10.10 Reporter: Bas Harenslak Assignee: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6882) Add an AWS SageMakerHook method for listing training jobs
Bas Harenslak created AIRFLOW-6882: -- Summary: Add an AWS SageMakerHook method for listing training jobs Key: AIRFLOW-6882 URL: https://issues.apache.org/jira/browse/AIRFLOW-6882 Project: Apache Airflow Issue Type: Bug Components: aws Affects Versions: 1.10.10 Reporter: Bas Harenslak Assignee: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6765) Test full DAGs with airflow dags test
[ https://issues.apache.org/jira/browse/AIRFLOW-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-6765: -- Assignee: Bas Harenslak > Test full DAGs with airflow dags test > - > > Key: AIRFLOW-6765 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6765 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > We now have a DebugExecutor for running a full DAG. Similar to airflow test > (airflow tasks test in 2.0), it would be nice to have a command "airflow dags > test", for running a full DAG with the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6765) Test full DAGs with airflow dags test
Bas Harenslak created AIRFLOW-6765: -- Summary: Test full DAGs with airflow dags test Key: AIRFLOW-6765 URL: https://issues.apache.org/jira/browse/AIRFLOW-6765 Project: Apache Airflow Issue Type: Bug Components: cli Affects Versions: 2.0.0 Reporter: Bas Harenslak We now have a DebugExecutor for running a full DAG. Similar to airflow test (airflow tasks test in 2.0), it would be nice to have a command "airflow dags test", for running a full DAG with the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4684) Make airflow/utils Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-4684. Resolution: Fixed > Make airflow/utils Pylint compatible > > > Key: AIRFLOW-4684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4684 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: kik >Priority: Major > > Fix all Pylint messages in airflow/utils. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4674) Make airflow/kubernetes Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-4674. Resolution: Fixed > Make airflow/kubernetes Pylint compatible > - > > Key: AIRFLOW-4674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4674 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: kik >Priority: Major > > Fix all Pylint messages in airflow/kubernetes. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4680) Make airflow/security Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-4680. Resolution: Fixed > Make airflow/security Pylint compatible > --- > > Key: AIRFLOW-4680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4680 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: kik >Priority: Major > > Fix all Pylint messages in airflow/security. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4666) Make airflow/contrib/sensors Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-4666. Resolution: Fixed > Make airflow/contrib/sensors Pylint compatible > -- > > Key: AIRFLOW-4666 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4666 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/contrib/sensors. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4671) Make airflow/executors Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-4671. Resolution: Fixed > Make airflow/executors Pylint compatible > > > Key: AIRFLOW-4671 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4671 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/executors. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6713) Make airflow/providers/amazon pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak resolved AIRFLOW-6713. Resolution: Fixed > Make airflow/providers/amazon pylint compatible > --- > > Key: AIRFLOW-6713 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6713 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > Make all code in airflow/providers/amazon pylint compatible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6713) Make airflow/providers/amazon pylint compatible
Bas Harenslak created AIRFLOW-6713: -- Summary: Make airflow/providers/amazon pylint compatible Key: AIRFLOW-6713 URL: https://issues.apache.org/jira/browse/AIRFLOW-6713 Project: Apache Airflow Issue Type: Sub-task Components: pylint Affects Versions: 2.0.0 Reporter: Bas Harenslak Make all code in airflow/providers/amazon pylint compatible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4375) Replace airflow.utils.file.TemporaryDirectory by native tempfile.TemporaryDirectory
[ https://issues.apache.org/jira/browse/AIRFLOW-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016023#comment-17016023 ] Bas Harenslak commented on AIRFLOW-4375: [~zhongjiajie] thanks for the notice, closed > Replace airflow.utils.file.TemporaryDirectory by native > tempfile.TemporaryDirectory > --- > > Key: AIRFLOW-4375 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4375 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Python 2 did not have a native temporarydirectory and I believe that's why > there is airflow.utils.file.TemporaryDirectory. > Since Python 3.2 this is supported by tempfile.TemporaryDirectory, so remove > the Airflow util and replace by the native Python tempfile.TemporaryDirectory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4375) Replace airflow.utils.file.TemporaryDirectory by native tempfile.TemporaryDirectory
[ https://issues.apache.org/jira/browse/AIRFLOW-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak closed AIRFLOW-4375. -- Resolution: Duplicate > Replace airflow.utils.file.TemporaryDirectory by native > tempfile.TemporaryDirectory > --- > > Key: AIRFLOW-4375 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4375 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Python 2 did not have a native temporarydirectory and I believe that's why > there is airflow.utils.file.TemporaryDirectory. > Since Python 3.2 this is supported by tempfile.TemporaryDirectory, so remove > the Airflow util and replace by the native Python tempfile.TemporaryDirectory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5880) Enforce unique task ids
Bas Harenslak created AIRFLOW-5880: -- Summary: Enforce unique task ids Key: AIRFLOW-5880 URL: https://issues.apache.org/jira/browse/AIRFLOW-5880 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak Currently, task with equal ids in a DAG simply overwrite each other and the last assigned task is saved. This can lead to unexpected behaviour and an exception should be raised. We've had a PendingDeprecationWarning on this for 4 years, time to get rid of it: [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5825) SageMakerEndpointOperator is not idempotent
[ https://issues.apache.org/jira/browse/AIRFLOW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5825: --- Description: The SageMakerEndpointOperator currently taken an argument "operation" with value "create"/"update" which determines whether to create or update a SageMaker endpoint. However this doesn't work in the following situation: * DAG run #1 create the endpoint (have to provide operation="create" here) * Following DAG runs will update the endpoint created by DAG run #1 (would have to edit DAG and set operation="update" here) Which should be a very valid use case IMO. The SageMakerEndpointOperator should check itself if an endpoint with name X already exists and overwrite it (configurable desired by the user). was: The SageMakerEndpointOperator currently taken an argument "operation" with value "create"/"update" which determines whether to create or update a SageMaker endpoint. However this doesn't work in the following situation: * DAG run #1 create the endpoint * Following DAG runs will update the endpoint created by DAG run #1 Which should be a very valid use case IMO. The SageMakerEndpointOperator should check itself if an endpoint with name X already exists and overwrite it (configurable desired by the user). > SageMakerEndpointOperator is not idempotent > --- > > Key: AIRFLOW-5825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5825 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0, 1.10.7 >Reporter: Bas Harenslak >Priority: Major > > The SageMakerEndpointOperator currently taken an argument "operation" with > value "create"/"update" which determines whether to create or update a > SageMaker endpoint. However this doesn't work in the following situation: > * DAG run #1 create the endpoint (have to provide operation="create" here) > * Following DAG runs will update the endpoint created by DAG run #1 (would > have to edit DAG and set operation="update" here) > Which should be a very valid use case IMO. > The SageMakerEndpointOperator should check itself if an endpoint with name X > already exists and overwrite it (configurable desired by the user). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5825) SageMakerEndpointOperator is not idempotent
Bas Harenslak created AIRFLOW-5825: -- Summary: SageMakerEndpointOperator is not idempotent Key: AIRFLOW-5825 URL: https://issues.apache.org/jira/browse/AIRFLOW-5825 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0, 1.10.7 Reporter: Bas Harenslak The SageMakerEndpointOperator currently taken an argument "operation" with value "create"/"update" which determines whether to create or update a SageMaker endpoint. However this doesn't work in the following situation: * DAG run #1 create the endpoint * Following DAG runs will update the endpoint created by DAG run #1 Which should be a very valid use case IMO. The SageMakerEndpointOperator should check itself if an endpoint with name X already exists and overwrite it (configurable desired by the user). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5788) Change import paths for AWS modules
[ https://issues.apache.org/jira/browse/AIRFLOW-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5788: --- Description: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs (docs/operators-and-hooks-ref.rst) * deprecation warning for old paths * migrating tests was: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs (docs/operators-and-hooks-ref.rst) * deprecation warning for old paths > Change import paths for AWS modules > --- > > Key: AIRFLOW-5788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5788 > Project: Apache Airflow > Issue Type: Sub-task > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > When migrating, think of the following: > * UPDATING.MD > * pylint_todo.txt > * docs (docs/operators-and-hooks-ref.rst) > * deprecation warning for old paths > * migrating tests -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5777) Migrate AWS DynamoDB to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5777 started by Bas Harenslak. -- > Migrate AWS DynamoDB to /providers/aws [AIP-21] > --- > > Key: AIRFLOW-5777 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5777 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5788) Change import paths for AWS modules
[ https://issues.apache.org/jira/browse/AIRFLOW-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5788: --- Description: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs * deprecation warning for old paths > Change import paths for AWS modules > --- > > Key: AIRFLOW-5788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5788 > Project: Apache Airflow > Issue Type: Sub-task > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > When migrating, think of the following: > * UPDATING.MD > * pylint_todo.txt > * docs > * deprecation warning for old paths -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5788) Change import paths for AWS modules
[ https://issues.apache.org/jira/browse/AIRFLOW-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5788: --- Description: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs (docs/operators-and-hooks-ref.rst) * deprecation warning for old paths * migrating tests * tests/test_core_to_contrib.py was: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs (docs/operators-and-hooks-ref.rst) * deprecation warning for old paths * migrating tests > Change import paths for AWS modules > --- > > Key: AIRFLOW-5788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5788 > Project: Apache Airflow > Issue Type: Sub-task > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > When migrating, think of the following: > * UPDATING.MD > * pylint_todo.txt > * docs (docs/operators-and-hooks-ref.rst) > * deprecation warning for old paths > * migrating tests > * tests/test_core_to_contrib.py -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5788) Change import paths for AWS modules
[ https://issues.apache.org/jira/browse/AIRFLOW-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5788: --- Description: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs (docs/operators-and-hooks-ref.rst) * deprecation warning for old paths was: When migrating, think of the following: * UPDATING.MD * pylint_todo.txt * docs * deprecation warning for old paths > Change import paths for AWS modules > --- > > Key: AIRFLOW-5788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5788 > Project: Apache Airflow > Issue Type: Sub-task > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > When migrating, think of the following: > * UPDATING.MD > * pylint_todo.txt > * docs (docs/operators-and-hooks-ref.rst) > * deprecation warning for old paths -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5790) Migrate AwsHook to /providers/aws
Bas Harenslak created AIRFLOW-5790: -- Summary: Migrate AwsHook to /providers/aws Key: AIRFLOW-5790 URL: https://issues.apache.org/jira/browse/AIRFLOW-5790 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5775) Migrate AWS Batch to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5775 started by Bas Harenslak. -- > Migrate AWS Batch to /providers/aws [AIP-21] > > > Key: AIRFLOW-5775 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5775 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5775) Migrate AWS Batch to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-5775: -- Assignee: Bas Harenslak > Migrate AWS Batch to /providers/aws [AIP-21] > > > Key: AIRFLOW-5775 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5775 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5773) Migrate AWS Athena components to /providers/aws
[ https://issues.apache.org/jira/browse/AIRFLOW-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5773 started by Bas Harenslak. -- > Migrate AWS Athena components to /providers/aws > --- > > Key: AIRFLOW-5773 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5773 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5787) Migrate AWS SQS to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5787: -- Summary: Migrate AWS SQS to /providers/aws [AIP-21] Key: AIRFLOW-5787 URL: https://issues.apache.org/jira/browse/AIRFLOW-5787 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5785) Migrate AWS SageMaker to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5785: -- Summary: Migrate AWS SageMaker to /providers/aws [AIP-21] Key: AIRFLOW-5785 URL: https://issues.apache.org/jira/browse/AIRFLOW-5785 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5779) Migrate AWS EMR to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5779: -- Summary: Migrate AWS EMR to /providers/aws [AIP-21] Key: AIRFLOW-5779 URL: https://issues.apache.org/jira/browse/AIRFLOW-5779 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5783) Migrate AWS Redshift to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5783: -- Summary: Migrate AWS Redshift to /providers/aws [AIP-21] Key: AIRFLOW-5783 URL: https://issues.apache.org/jira/browse/AIRFLOW-5783 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5773) Migrate AWS Athena components to /providers/aws
[ https://issues.apache.org/jira/browse/AIRFLOW-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-5773: -- Assignee: Bas Harenslak > Migrate AWS Athena components to /providers/aws > --- > > Key: AIRFLOW-5773 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5773 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5788) Change import paths for AWS modules
Bas Harenslak created AIRFLOW-5788: -- Summary: Change import paths for AWS modules Key: AIRFLOW-5788 URL: https://issues.apache.org/jira/browse/AIRFLOW-5788 Project: Apache Airflow Issue Type: Sub-task Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5786: -- Summary: Migrate AWS SNS to /providers/aws [AIP-21] Key: AIRFLOW-5786 URL: https://issues.apache.org/jira/browse/AIRFLOW-5786 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5778) Migrate AWS ECS to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5778: -- Summary: Migrate AWS ECS to /providers/aws [AIP-21] Key: AIRFLOW-5778 URL: https://issues.apache.org/jira/browse/AIRFLOW-5778 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5780) Migrate AWS Glue to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5780: -- Summary: Migrate AWS Glue to /providers/aws [AIP-21] Key: AIRFLOW-5780 URL: https://issues.apache.org/jira/browse/AIRFLOW-5780 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5784) Migrate AWS S3 to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5784: -- Summary: Migrate AWS S3 to /providers/aws [AIP-21] Key: AIRFLOW-5784 URL: https://issues.apache.org/jira/browse/AIRFLOW-5784 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5781) Migrate AWS Kinesis to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5781: -- Summary: Migrate AWS Kinesis to /providers/aws [AIP-21] Key: AIRFLOW-5781 URL: https://issues.apache.org/jira/browse/AIRFLOW-5781 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5782) Migrate AWS Lambda to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5782: -- Summary: Migrate AWS Lambda to /providers/aws [AIP-21] Key: AIRFLOW-5782 URL: https://issues.apache.org/jira/browse/AIRFLOW-5782 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5776) Migrate AWS Cloudwatch logs to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5776: -- Summary: Migrate AWS Cloudwatch logs to /providers/aws [AIP-21] Key: AIRFLOW-5776 URL: https://issues.apache.org/jira/browse/AIRFLOW-5776 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5775) Migrate AWS Batch to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5775: -- Summary: Migrate AWS Batch to /providers/aws [AIP-21] Key: AIRFLOW-5775 URL: https://issues.apache.org/jira/browse/AIRFLOW-5775 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5777) Migrate AWS DynamoDB to /providers/aws [AIP-21]
Bas Harenslak created AIRFLOW-5777: -- Summary: Migrate AWS DynamoDB to /providers/aws [AIP-21] Key: AIRFLOW-5777 URL: https://issues.apache.org/jira/browse/AIRFLOW-5777 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5773) Migrate AWS Athena components to /providers/aws
Bas Harenslak created AIRFLOW-5773: -- Summary: Migrate AWS Athena components to /providers/aws Key: AIRFLOW-5773 URL: https://issues.apache.org/jira/browse/AIRFLOW-5773 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5733) Move all AWS components to its own submodule
[
https://issues.apache.org/jira/browse/AIRFLOW-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bas Harenslak reassigned AIRFLOW-5733:
--
Assignee: Bas Harenslak
> Move all AWS components to its own submodule
>
>
> Key: AIRFLOW-5733
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5733
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>
> As voted for in
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3\{aws/azure/gcp}_*],
> and in similar fashion to all GCP components, which are now stored under
> /airflow/gcp, we should move all AWS components to /airflow/aws.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5734) Move all Azure components to its own submodule
Bas Harenslak created AIRFLOW-5734:
--
Summary: Move all Azure components to its own submodule
Key: AIRFLOW-5734
URL: https://issues.apache.org/jira/browse/AIRFLOW-5734
Project: Apache Airflow
Issue Type: Improvement
Components: core
Affects Versions: 2.0.0
Reporter: Bas Harenslak
As voted for in
[https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3\{aws/azure/gcp}_*],
and in similar fashion to all GCP components, which are now stored under
/airflow/gcp, we should move all Azure components to /airflow/azure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5733) Move all AWS components to its own submodule
Bas Harenslak created AIRFLOW-5733:
--
Summary: Move all AWS components to its own submodule
Key: AIRFLOW-5733
URL: https://issues.apache.org/jira/browse/AIRFLOW-5733
Project: Apache Airflow
Issue Type: Improvement
Components: aws
Affects Versions: 2.0.0
Reporter: Bas Harenslak
As voted for in
[https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3\{aws/azure/gcp}_*],
and in similar fashion to all GCP components, which are now stored under
/airflow/gcp, we should move all AWS components to /airflow/aws.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5729) Make InputDataConfig in SageMakerTrainingOperator optional
Bas Harenslak created AIRFLOW-5729: -- Summary: Make InputDataConfig in SageMakerTrainingOperator optional Key: AIRFLOW-5729 URL: https://issues.apache.org/jira/browse/AIRFLOW-5729 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 1.10.5, 2.0.0 Reporter: Bas Harenslak Currently the SageMakerTrainingOperator will fail when no config key InputDataConfig is supplied, however this is not a required property, see https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html#SageMaker-CreateTrainingJob-request-InputDataConfig. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5647) Airflow UI should also display dag_concurrency reached
Bas Harenslak created AIRFLOW-5647: -- Summary: Airflow UI should also display dag_concurrency reached Key: AIRFLOW-5647 URL: https://issues.apache.org/jira/browse/AIRFLOW-5647 Project: Apache Airflow Issue Type: Improvement Components: ui Affects Versions: 1.10.5 Reporter: Bas Harenslak Currently, in the main view, the schedule column box is highlighted in red if the max. number of DAG runs is achieved. In this case no more DAG runs can be started until a DAG run completes. I think it should also display in red when the dag_concurrency (i.e. max concurrent tasks) is achieved. In this case also, no more tasks can be started until a task completes. However there is currently nothing in the UI showing that (currently running 1.10.5). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5644) Simplify TriggerDagRunOperator usage
Bas Harenslak created AIRFLOW-5644: -- Summary: Simplify TriggerDagRunOperator usage Key: AIRFLOW-5644 URL: https://issues.apache.org/jira/browse/AIRFLOW-5644 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 2.0.0 Reporter: Bas Harenslak The TriggerDagRunOperator usage is rather odd at the moment, especially the way to pass a conf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5307) Move the BaseOperator to the operators package
[ https://issues.apache.org/jira/browse/AIRFLOW-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5307: --- Description: The BaseOperator currently resides in /airflow/models but has never been a database model, i.e. it is not stored in a database table. I suggest to move it to its logical place, the operators package. > Move the BaseOperator to the operators package > -- > > Key: AIRFLOW-5307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5307 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > The BaseOperator currently resides in /airflow/models but has never been a > database model, i.e. it is not stored in a database table. I suggest to move > it to its logical place, the operators package. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5307) Move the BaseOperator to the operators package
Bas Harenslak created AIRFLOW-5307: -- Summary: Move the BaseOperator to the operators package Key: AIRFLOW-5307 URL: https://issues.apache.org/jira/browse/AIRFLOW-5307 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-4712) Make tests/ti_deps Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4712: -- Assignee: Bas Harenslak > Make tests/ti_deps Pylint compatible > > > Key: AIRFLOW-4712 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4712 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in tests/ti_deps. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (AIRFLOW-5268) Rectify DAG naming conventions
[ https://issues.apache.org/jira/browse/AIRFLOW-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5268: --- Affects Version/s: (was: 1.10.5) > Rectify DAG naming conventions > -- > > Key: AIRFLOW-5268 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5268 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5268) Rectify DAG naming conventions
Bas Harenslak created AIRFLOW-5268: -- Summary: Rectify DAG naming conventions Key: AIRFLOW-5268 URL: https://issues.apache.org/jira/browse/AIRFLOW-5268 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0, 1.10.5 Reporter: Bas Harenslak -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5241) Make all test class names consistent by starting with Test
Bas Harenslak created AIRFLOW-5241: -- Summary: Make all test class names consistent by starting with Test Key: AIRFLOW-5241 URL: https://issues.apache.org/jira/browse/AIRFLOW-5241 Project: Apache Airflow Issue Type: Improvement Components: tests Affects Versions: 2.0.0, 1.10.5 Reporter: Bas Harenslak -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5230) Refactor template file loading functions
Bas Harenslak created AIRFLOW-5230: -- Summary: Refactor template file loading functions Key: AIRFLOW-5230 URL: https://issues.apache.org/jira/browse/AIRFLOW-5230 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5163) Default do_xcom_push to False
[ https://issues.apache.org/jira/browse/AIRFLOW-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5163: --- Description: By default BaseOperator.do_xcom_push is set to True. This is most of the time not desirable and must be explicitly turned off. It makes more sense to set it to False by default. (was: By default BaseOperator.do_xcom_push is set to True. This is most of the time not desirable and must be explicitly turned off. It makes more sense to set it to False by default. Also, the name is ambiguous and e.g. "xcom_push_result" would be more meaningful) > Default do_xcom_push to False > - > > Key: AIRFLOW-5163 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5163 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > By default BaseOperator.do_xcom_push is set to True. This is most of the time > not desirable and must be explicitly turned off. It makes more sense to set > it to False by default. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5163) Default do_xcom_push to False and give meaningful name
[ https://issues.apache.org/jira/browse/AIRFLOW-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5163: --- Description: By default BaseOperator.do_xcom_push is set to True. This is most of the time not desirable and must be explicitly turned off. It makes more sense to set it to False by default. Also, the name is ambiguous and e.g. "xcom_push_result" would be more meaningful was:By default BaseOperator.do_xcom_push is set to True. This is most of the time not desirable and must be explicitly turned off. It makes more sense to set it to False by default. > Default do_xcom_push to False and give meaningful name > -- > > Key: AIRFLOW-5163 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5163 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > By default BaseOperator.do_xcom_push is set to True. This is most of the time > not desirable and must be explicitly turned off. It makes more sense to set > it to False by default. > Also, the name is ambiguous and e.g. "xcom_push_result" would be more > meaningful -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5163) Default do_xcom_push to False and give meaningful name
[ https://issues.apache.org/jira/browse/AIRFLOW-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5163: --- Summary: Default do_xcom_push to False and give meaningful name (was: Default do_xcom_push to False) > Default do_xcom_push to False and give meaningful name > -- > > Key: AIRFLOW-5163 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5163 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > By default BaseOperator.do_xcom_push is set to True. This is most of the time > not desirable and must be explicitly turned off. It makes more sense to set > it to False by default. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5163) Default do_xcom_push to False
[ https://issues.apache.org/jira/browse/AIRFLOW-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-5163: --- Summary: Default do_xcom_push to False (was: Default do_xcom_push to False and give meaningful name) > Default do_xcom_push to False > - > > Key: AIRFLOW-5163 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5163 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > By default BaseOperator.do_xcom_push is set to True. This is most of the time > not desirable and must be explicitly turned off. It makes more sense to set > it to False by default. > Also, the name is ambiguous and e.g. "xcom_push_result" would be more > meaningful -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5163) Default do_xcom_push to False
Bas Harenslak created AIRFLOW-5163: -- Summary: Default do_xcom_push to False Key: AIRFLOW-5163 URL: https://issues.apache.org/jira/browse/AIRFLOW-5163 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 2.0.0 Reporter: Bas Harenslak By default BaseOperator.do_xcom_push is set to True. This is most of the time not desirable and must be explicitly turned off. It makes more sense to set it to False by default. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5160) Remove DAG count test
Bas Harenslak created AIRFLOW-5160: -- Summary: Remove DAG count test Key: AIRFLOW-5160 URL: https://issues.apache.org/jira/browse/AIRFLOW-5160 Project: Apache Airflow Issue Type: Improvement Components: tests Affects Versions: 2.0.0, 1.10.5 Reporter: Bas Harenslak The test_import_examples test asserts the number of example DAGs. It does not verify any behaviour and fails when example DAGs are added/removed. It doesn't test behaviour so I suggest to remove it. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5105) Move tests out of tests/core.py
Bas Harenslak created AIRFLOW-5105: -- Summary: Move tests out of tests/core.py Key: AIRFLOW-5105 URL: https://issues.apache.org/jira/browse/AIRFLOW-5105 Project: Apache Airflow Issue Type: Improvement Components: tests Affects Versions: 2.0.0, 1.10.5 Reporter: Bas Harenslak There are >2000 lines of tests in tests/core.py. The pattern followed in the Airflow code base is to create a test file named test_[file to test]. There is no core.py in airflow/ so move all tests to the proper location. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (AIRFLOW-4217) Remove six package in project
[ https://issues.apache.org/jira/browse/AIRFLOW-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4217: -- Assignee: Bas Harenslak (was: Chao-Han Tsai) > Remove six package in project > - > > Key: AIRFLOW-4217 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4217 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: zhongjiajie >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > Remove all six package in Airflow -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (AIRFLOW-4992) Replace backports.configparser in favour of Python 3 native configparser
[ https://issues.apache.org/jira/browse/AIRFLOW-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4992: -- Assignee: Bas Harenslak > Replace backports.configparser in favour of Python 3 native configparser > > > Key: AIRFLOW-4992 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4992 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-4992) Replace backports.configparser in favour of Python 3 native configparser
Bas Harenslak created AIRFLOW-4992: -- Summary: Replace backports.configparser in favour of Python 3 native configparser Key: AIRFLOW-4992 URL: https://issues.apache.org/jira/browse/AIRFLOW-4992 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-4963) Avoid recreating task context
Bas Harenslak created AIRFLOW-4963: -- Summary: Avoid recreating task context Key: AIRFLOW-4963 URL: https://issues.apache.org/jira/browse/AIRFLOW-4963 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.5 Reporter: Bas Harenslak The task context is created by taskinstance.get_template_context() and called twice via _run_raw_task: # context = self.get_template_context() # 2nd time in render_template(), instead of passing it to render_template() This is costly (lots of variables initialized, DB session, datetime calculations) and can be avoided. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-4963) Avoid recreating task context
[ https://issues.apache.org/jira/browse/AIRFLOW-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-4963: --- Labels: performance (was: ) > Avoid recreating task context > - > > Key: AIRFLOW-4963 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4963 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.5 >Reporter: Bas Harenslak >Priority: Major > Labels: performance > > The task context is created by taskinstance.get_template_context() and called > twice via _run_raw_task: > # context = self.get_template_context() > # 2nd time in render_template(), instead of passing it to render_template() > This is costly (lots of variables initialized, DB session, datetime > calculations) and can be avoided. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (AIRFLOW-4678) Make airflow/models Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4678: -- Assignee: Bas Harenslak > Make airflow/models Pylint compatible > - > > Key: AIRFLOW-4678 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4678 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/models. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4835) Refactor template rendering functions
[ https://issues.apache.org/jira/browse/AIRFLOW-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-4835: --- Summary: Refactor template rendering functions (was: Refactor templating rendering functions) > Refactor template rendering functions > - > > Key: AIRFLOW-4835 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4835 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4835) Refactor templating rendering functions
Bas Harenslak created AIRFLOW-4835: -- Summary: Refactor templating rendering functions Key: AIRFLOW-4835 URL: https://issues.apache.org/jira/browse/AIRFLOW-4835 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4689) Make setup.py Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4689 started by Bas Harenslak. -- > Make setup.py Pylint compatible > --- > > Key: AIRFLOW-4689 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4689 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in setup.py. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4689) Make setup.py Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4689: -- Assignee: Bas Harenslak > Make setup.py Pylint compatible > --- > > Key: AIRFLOW-4689 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4689 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in setup.py. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4751) Future Pylint TODOs
Bas Harenslak created AIRFLOW-4751: -- Summary: Future Pylint TODOs Key: AIRFLOW-4751 URL: https://issues.apache.org/jira/browse/AIRFLOW-4751 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak This issue keeps track of configurations we would to make with Pylint but are currently not possible. See [https://github.com/apache/airflow/pull/5238] for details about initial investigation about all Pylint errors. * Pylint raises "pointless-statement" on the bitshift operators, mostly on the example DAGs. In order to not pollute all example dag files, we disabled the message globally. However, there were some of such messages outside the example DAGs for valid reasons, so would be nice to disable this specific message on the example_dags folder, which is currently not possible. Pylint has a project to track this functionality: [https://github.com/PyCQA/pylint/projects/1] * A module docstring is sometimes obsolete while other docstring are typically always desired. Currently Pylint does not allow configuring separate docstrings. Issue: [https://github.com/PyCQA/pylint/issues/1164] * -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4725: -- Assignee: Bas Harenslak > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4725) Make setup.py run without errors/warnings
Bas Harenslak created AIRFLOW-4725: -- Summary: Make setup.py run without errors/warnings Key: AIRFLOW-4725 URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak Currently running "python setup.py --help-commands" (on master) throws a warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4364) Integrate Pylint
[ https://issues.apache.org/jira/browse/AIRFLOW-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-4364: --- Description: After a [vote on the mailing list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] everybody voted for pylint integration. Making the whole project Pylint compatible is a lot of work and big change. Therefore we split up all the work in subissues under this issue. The approach is as follows: All files are currently blacklisted from Pylint. The blacklist is stored in scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on the blacklist. Once you start on an issue: # (running scripts/ci/ci_pylint.sh on master should produce no messages) # Remove the files mentioned in your issue from the blacklist # Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files # Fix all messages and create PR *Why a separate blacklist file and not use Pylint's --ignore-pattern to ignore files?* --ignore-pattern only works on base filenames, not paths. *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* Creating a list of non-overlapping patterns proved difficult, this was the pragmatic solution. *Rule X is too strict. Can we disable it?* In the first PR ([https://github.com/apache/airflow/pull/5238]) we made a choice on every error found on Airflow master back then. While at occasions it might seem harsh to be strict on the code, Airflow is an open source project with many contributors from all over the world. Others read the code without the thought process you put into the code and it helps to have e.g. descriptive variable names, docstrings and sticking to Python conventions. This helps the collaboration between everybody and even your future self. Typically, this question suggests one is trying to lower the boundaries. If you believe there is a valid reason for doing so, please add it to the PR and explain the reason. was: After a [vote on the mailing list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] everybody voted for pylint integration. Making the whole project Pylint compatible is a lot of work and big change. Therefore we split up all the work in subissues under this issue. The approach is as follows: All files are currently blacklisted from Pylint. The blacklist is stored in scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on the blacklist. Once you start on an issue: # (running scripts/ci/ci_pylint.sh on master should produce no messages) # Remove the files mentioned in your issue from the blacklist # Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files # Fix all messages and create PR *Why a separate blacklist file and not use Pylint's --ignore-pattern to ignore files?* --ignore-pattern only works on base filenames, not paths. *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* Creating a list of non-overlapping patterns proved difficult, this was the pragmatic solution. *Rule X is too strict. Can we disable it?* dsT > Integrate Pylint > > > Key: AIRFLOW-4364 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4364 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > After a [vote on the mailing > list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] > everybody voted for pylint integration. > Making the whole project Pylint compatible is a lot of work and big change. > Therefore we split up all the work in subissues under this issue. The > approach is as follows: > All files are currently blacklisted from Pylint. The blacklist is stored in > scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on > the blacklist. Once you start on an issue: > # (running scripts/ci/ci_pylint.sh on master should produce no messages) > # Remove the files mentioned in your issue from the blacklist > # Run scripts/ci/ci_pylint.sh to see all messages on the no longer > blacklisted files > # Fix all messages and create PR > *Why a separate blacklist file and not use Pylint's --ignore-pattern to > ignore files?* > --ignore-pattern only works on base filenames, not paths. > *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* > Creating a list of non-overlapping patterns proved difficult, this was the > pragmatic solution. > *Rule X is too strict. Can we disable it?* > In the first PR ([https://github.com/apache/airflow/pull/5238]) we made a > choice on every
[jira] [Updated] (AIRFLOW-4364) Integrate Pylint
[ https://issues.apache.org/jira/browse/AIRFLOW-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-4364: --- Description: After a [vote on the mailing list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] everybody voted for pylint integration. Making the whole project Pylint compatible is a lot of work and big change. Therefore we split up all the work in subissues under this issue. The approach is as follows: All files are currently blacklisted from Pylint. The blacklist is stored in scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on the blacklist. Once you start on an issue: # (running scripts/ci/ci_pylint.sh on master should produce no messages) # Remove the files mentioned in your issue from the blacklist # Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files # Fix all messages and create PR *Why a separate blacklist file and not use Pylint's --ignore-pattern to ignore files?* --ignore-pattern only works on base filenames, not paths. *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* Creating a list of non-overlapping patterns proved difficult, this was the pragmatic solution. *Rule X is too strict. Can we disable it?* dsT was: After a [vote on the mailing list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] everybody voted for pylint integration. Making the whole project Pylint compatible is a lot of work and big change. Therefore we split up all the work in subissues under this issue. The approach is as follows: All files are currently blacklisted from Pylint. The blacklist is stored in scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on the blacklist. Once you start on an issue: # (running scripts/ci/ci_pylint.sh on master should produce no messages) # Remove the files mentioned in your issue from the blacklist # Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files # Fix all messages and create PR *Why a separate blacklist file and not use Pylint's --ignore-pattern to ignore files?* --ignore-pattern only works on base filenames, not paths. *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* Creating a list of non-overlapping patterns proved difficult, this was the pragmatic solution. > Integrate Pylint > > > Key: AIRFLOW-4364 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4364 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > > After a [vote on the mailing > list|https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba@%3Cdev.airflow.apache.org%3E] > everybody voted for pylint integration. > Making the whole project Pylint compatible is a lot of work and big change. > Therefore we split up all the work in subissues under this issue. The > approach is as follows: > All files are currently blacklisted from Pylint. The blacklist is stored in > scripts/ci/pylint_todo.txt. Every subissue relates to one or more files on > the blacklist. Once you start on an issue: > # (running scripts/ci/ci_pylint.sh on master should produce no messages) > # Remove the files mentioned in your issue from the blacklist > # Run scripts/ci/ci_pylint.sh to see all messages on the no longer > blacklisted files > # Fix all messages and create PR > *Why a separate blacklist file and not use Pylint's --ignore-pattern to > ignore files?* > --ignore-pattern only works on base filenames, not paths. > *Why don't you blacklist patterns, where 1 line relates to 1 JIRA issue?* > Creating a list of non-overlapping patterns proved difficult, this was the > pragmatic solution. > *Rule X is too strict. Can we disable it?* > dsT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4669) Make airflow/dag Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4669 started by Bas Harenslak. -- > Make airflow/dag Pylint compatible > -- > > Key: AIRFLOW-4669 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4669 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/dag. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4670) Make airflow/example_dags Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4670 started by Bas Harenslak. -- > Make airflow/example_dags Pylint compatible > --- > > Key: AIRFLOW-4670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4670 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/example_dags. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4671) Make airflow/executors Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4671: -- Assignee: Bas Harenslak > Make airflow/executors Pylint compatible > > > Key: AIRFLOW-4671 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4671 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/executors. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4670) Make airflow/example_dags Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4670: -- Assignee: Bas Harenslak > Make airflow/example_dags Pylint compatible > --- > > Key: AIRFLOW-4670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4670 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/example_dags. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4723) Remove the deprecated pass-through conf methods
[
https://issues.apache.org/jira/browse/AIRFLOW-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bas Harenslak updated AIRFLOW-4723:
---
Description:
This has been deprecated since 2016 (airflow/configuration.py):
{code}
# Historical convenience functions to access config entries
load_test_config = conf.load_test_config
get = conf.get
getboolean = conf.getboolean
getfloat = conf.getfloat
getint = conf.getint
getsection = conf.getsection
has_option = conf.has_option
remove_option = conf.remove_option
as_dict = conf.as_dict
set = conf.set # noqa
for func in [load_test_config, get, getboolean, getfloat, getint, has_option,
remove_option, as_dict, set]:
deprecated(
func,
"Accessing configuration method '{f.__name__}' directly from "
"the configuration module is deprecated. Please access the "
"configuration from the 'configuration.conf' object via "
"'conf.{f.__name__}'".format(f=func)){code}
Time to remove it and unify the way to access conf.
> Remove the deprecated pass-through conf methods
> ---
>
> Key: AIRFLOW-4723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4723
> Project: Apache Airflow
> Issue Type: Improvement
> Components: configuration
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>
> This has been deprecated since 2016 (airflow/configuration.py):
> {code}
> # Historical convenience functions to access config entries
> load_test_config = conf.load_test_config
> get = conf.get
> getboolean = conf.getboolean
> getfloat = conf.getfloat
> getint = conf.getint
> getsection = conf.getsection
> has_option = conf.has_option
> remove_option = conf.remove_option
> as_dict = conf.as_dict
> set = conf.set # noqa
> for func in [load_test_config, get, getboolean, getfloat, getint, has_option,
> remove_option, as_dict, set]:
> deprecated(
> func,
> "Accessing configuration method '{f.__name__}' directly from "
> "the configuration module is deprecated. Please access the "
> "configuration from the 'configuration.conf' object via "
> "'conf.{f.__name__}'".format(f=func)){code}
> Time to remove it and unify the way to access conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4723) Remove the deprecated pass-through conf methods
[ https://issues.apache.org/jira/browse/AIRFLOW-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4723: -- Assignee: Bas Harenslak > Remove the deprecated pass-through conf methods > --- > > Key: AIRFLOW-4723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4723 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4723) Remove the deprecated pass-through conf methods
Bas Harenslak created AIRFLOW-4723: -- Summary: Remove the deprecated pass-through conf methods Key: AIRFLOW-4723 URL: https://issues.apache.org/jira/browse/AIRFLOW-4723 Project: Apache Airflow Issue Type: Improvement Components: configuration Affects Versions: 2.0.0 Reporter: Bas Harenslak -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4721) Remove all import builtins imports
[ https://issues.apache.org/jira/browse/AIRFLOW-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4721: -- Assignee: Bas Harenslak > Remove all import builtins imports > -- > > Key: AIRFLOW-4721 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4721 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > There's lots of from builtins import ... imports in the codebase. These are > all the result of Python 2/3 compatibility and should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4721) Remove all import builtins imports
Bas Harenslak created AIRFLOW-4721: -- Summary: Remove all import builtins imports Key: AIRFLOW-4721 URL: https://issues.apache.org/jira/browse/AIRFLOW-4721 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Bas Harenslak There's lots of from builtins import ... imports in the codebase. These are all the result of Python 2/3 compatibility and should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4657) Make airflow/[a-zA-Z_]*.py Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4657: -- Assignee: Bas Harenslak > Make airflow/[a-zA-Z_]*.py Pylint compatible > > > Key: AIRFLOW-4657 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4657 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/[a-zA-Z_]*.py. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4669) Make airflow/dag Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak reassigned AIRFLOW-4669: -- Assignee: Bas Harenslak > Make airflow/dag Pylint compatible > -- > > Key: AIRFLOW-4669 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4669 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Fix all Pylint messages in airflow/dag. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4661) Make airflow/config_templates Pylint compatible
Bas Harenslak created AIRFLOW-4661: -- Summary: Make airflow/config_templates Pylint compatible Key: AIRFLOW-4661 URL: https://issues.apache.org/jira/browse/AIRFLOW-4661 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in airflow/config_templates. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4669) Make airflow/dag Pylint compatible
Bas Harenslak created AIRFLOW-4669: -- Summary: Make airflow/dag Pylint compatible Key: AIRFLOW-4669 URL: https://issues.apache.org/jira/browse/AIRFLOW-4669 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in airflow/dag. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4702) Make tests/macros Pylint compatible
Bas Harenslak created AIRFLOW-4702: -- Summary: Make tests/macros Pylint compatible Key: AIRFLOW-4702 URL: https://issues.apache.org/jira/browse/AIRFLOW-4702 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in tests/macros. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4709) Make tests/sensors Pylint compatible
Bas Harenslak created AIRFLOW-4709: -- Summary: Make tests/sensors Pylint compatible Key: AIRFLOW-4709 URL: https://issues.apache.org/jira/browse/AIRFLOW-4709 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in tests/sensors. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4703) Make tests/migrations Pylint compatible
Bas Harenslak created AIRFLOW-4703: -- Summary: Make tests/migrations Pylint compatible Key: AIRFLOW-4703 URL: https://issues.apache.org/jira/browse/AIRFLOW-4703 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in tests/migrations. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4682) Make airflow/task Pylint compatible
Bas Harenslak created AIRFLOW-4682: -- Summary: Make airflow/task Pylint compatible Key: AIRFLOW-4682 URL: https://issues.apache.org/jira/browse/AIRFLOW-4682 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in airflow/task. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4686) Make dags Pylint compatible
Bas Harenslak created AIRFLOW-4686: -- Summary: Make dags Pylint compatible Key: AIRFLOW-4686 URL: https://issues.apache.org/jira/browse/AIRFLOW-4686 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in dags. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4708) Make tests/security Pylint compatible
Bas Harenslak created AIRFLOW-4708: -- Summary: Make tests/security Pylint compatible Key: AIRFLOW-4708 URL: https://issues.apache.org/jira/browse/AIRFLOW-4708 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Bas Harenslak Fix all Pylint messages in tests/security. To start; running scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the files mentioned in your issue from the blacklist. (2) Run scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted files. (3) Fix all messages and create PR. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
