[jira] [Resolved] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-7068. --- Fix Version/s: 2.0.0 Resolution: Fixed > Create EC2 Hook, Operator and Sensor > > > Key: AIRFLOW-7068 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7068 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Mustafa Gök >Assignee: Mustafa Gök >Priority: Minor > Fix For: 2.0.0 > > > * New hook to interact with AWS EC2 Service. > * New operator to manage AWS EC2 instance and to change instance state by > applying given operation using boto3. > * New sensor to check the state of the AWS EC2 instance until state of the > instance become equal to the target state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-4675) Make airflow/lineage Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall reassigned AIRFLOW-4675: - Assignee: Felix Uellendall (was: enrica) > Make airflow/lineage Pylint compatible > -- > > Key: AIRFLOW-4675 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4675 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Felix Uellendall >Priority: Major > > Fix all Pylint messages in airflow/lineage. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6959) Use NULL as dag.description default value and change related UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6959. --- Fix Version/s: 2.0.0 Resolution: Fixed > Use NULL as dag.description default value and change related UI > --- > > Key: AIRFLOW-6959 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6959 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG, database, ui >Affects Versions: 1.10.9 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5256) Related pylint changes for common licences in python files
[ https://issues.apache.org/jira/browse/AIRFLOW-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5256: -- Fix Version/s: (was: 1.10.6) 2.0.0 > Related pylint changes for common licences in python files > -- > > Key: AIRFLOW-5256 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5256 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-4672) Make airflow/hooks Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall reassigned AIRFLOW-4672: - Assignee: Felix Uellendall (was: Omkar) > Make airflow/hooks Pylint compatible > > > Key: AIRFLOW-4672 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4672 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Felix Uellendall >Priority: Major > > Fix all Pylint messages in airflow/hooks. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-4679) Make airflow/operators Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall reassigned AIRFLOW-4679: - Assignee: Felix Uellendall > Make airflow/operators Pylint compatible > > > Key: AIRFLOW-4679 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4679 > Project: Apache Airflow > Issue Type: Sub-task > Components: pylint >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Felix Uellendall >Priority: Major > > Fix all Pylint messages in airflow/operators. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5062) Allow ACL header in S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall closed AIRFLOW-5062. - Resolution: Duplicate > Allow ACL header in S3 Hook > --- > > Key: AIRFLOW-5062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5062 > Project: Apache Airflow > Issue Type: New Feature > Components: aws >Affects Versions: 1.10.3 >Reporter: Lester kim >Assignee: Lester kim >Priority: Critical > > Definition of Done > * Add ACL field in S3Hook when uploading a file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reopened AIRFLOW-4175:
---
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall closed AIRFLOW-4175.
-
Resolution: Fixed
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5922. --- Fix Version/s: 2.0.0 Resolution: Fixed > Add option to specify the mysql client library used in MySqlHook > > > Key: AIRFLOW-5922 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5922 > Project: Apache Airflow > Issue Type: New Feature > Components: dependencies, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-2325) Task logging with AWS Cloud watch
[ https://issues.apache.org/jira/browse/AIRFLOW-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-2325. --- Fix Version/s: 2.0.0 Resolution: Fixed > Task logging with AWS Cloud watch > - > > Key: AIRFLOW-2325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2325 > Project: Apache Airflow > Issue Type: New Feature > Components: logging >Reporter: Fang-Pen Lin >Assignee: Daniel Hegberg >Priority: Minor > Fix For: 2.0.0 > > > In many cases, it's ideal to use remote logging while running Airflow in > production, as the worker could be easily scale down or scale up. Or the > worker is running in containers, where the local storage is not meant to be > there forever. In that case, the S3 task logging handler could be used > [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/s3_task_handler.py] > However, it comes with drawback. S3 logging handler only uploads the log when > the task completed or failed. For long running tasks, it's hard to know > what's going on with the process until it finishes. > To make more real-time logging, I built a logging handler based on AWS > CloudWatch. It uses a third party python package `watchtower` > > [https://github.com/kislyuk/watchtower/tree/master/watchtower] > > I created a PR here [https://github.com/apache/incubator-airflow/pull/3229], > basically I just copy-pasted the code I wrote for my own project, it works > fine with 1.9 release, but never tested with master branch. Also, there is a > bug in watchtower causing task runner to hang forever when it completes. I > created an issue in their repo > [https://github.com/kislyuk/watchtower/issues/57] > And a PR for addressing that issue > [https://github.com/kislyuk/watchtower/pull/58] > > The PR is still far from ready to be reviewed, but I just want to get some > feedback before I spend more time on it. I would like to see if youguys want > this cloudwatch handler goes into the main repo, or do youguys prefer it to > be a standalone third-party module. If it's that case, I can close this > ticket and create a standalone repo on my own. If the PR is welcome, then I > can spend more time on polishing it based on your feedback, add tests / > documents and other stuff. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5908) Add download_file to S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5908. --- Fix Version/s: 2.0.0 Resolution: Fixed > Add download_file to S3 Hook > > > Key: AIRFLOW-5908 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5924. --- Resolution: Fixed > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6830) AWS SNS Hook and Operator should both support Subject and MessageAttributes
[ https://issues.apache.org/jira/browse/AIRFLOW-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6830. --- Fix Version/s: 1.10.10 Resolution: Fixed > AWS SNS Hook and Operator should both support Subject and MessageAttributes > --- > > Key: AIRFLOW-6830 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6830 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, hooks, operators >Affects Versions: 1.10.9 >Reporter: Luyao Yang >Assignee: Luyao Yang >Priority: Minor > Labels: AWS > Fix For: 1.10.10 > > > Support for the Subject field for AWS SNS was added to AwsSnsHook in > [AIRFLOW-5082|https://issues.apache.org/jira/browse/AIRFLOW-5082]. > MessageAttributes is another field that can be passed with any message type > and should be supported. > Furthermore, both fields should be available through SnsPublishOperator as > well so we don't have to write a Python function whenever one of these fields > is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6616) Add Tableau Hook and Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043380#comment-17043380 ] Felix Uellendall commented on AIRFLOW-6616: --- That would be great! :) > Add Tableau Hook and Operators > -- > > Key: AIRFLOW-6616 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6616 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 2.0.0 >Reporter: Rok Mihevc >Priority: Minor > Fix For: 2.0.0 > > > Tableau (tableau.com) provides a python client > (github.com/tableau/server-client-python) that enables automation of certain > tasks (e.g. report sending). > Addition of Tableau hook and operators will enable using Tableau to generate > and send as steps in a wider ETL process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6825) when build from source and use rbac=True UI is stuck
[ https://issues.apache.org/jira/browse/AIRFLOW-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall closed AIRFLOW-6825. - Resolution: Fixed I am closing this for now. Let me know if you experience further issues. Thanks, Felix > when build from source and use rbac=True UI is stuck > > > Key: AIRFLOW-6825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6825 > Project: Apache Airflow > Issue Type: Bug > Components: authentication >Affects Versions: 1.10.7 >Reporter: Rajender Reddy Kompally >Priority: Blocker > Attachments: airflow_rbac_issue.png > > > When we build airflow 1.10.7 from source and run using rbac = True option UI > is stuck and an't go to any view tabs. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6825) when build from source and use rbac=True UI is stuck
[ https://issues.apache.org/jira/browse/AIRFLOW-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043192#comment-17043192 ] Felix Uellendall commented on AIRFLOW-6825: --- Hey [~rreddy3], It looks like you did not build the bundled js / css. If you are building from source you als need to [generate the bundles files with yarn|https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#generate-bundled-files-with-yarn] Start from here: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#nodejs-environment-setup Best Regards, Felix > when build from source and use rbac=True UI is stuck > > > Key: AIRFLOW-6825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6825 > Project: Apache Airflow > Issue Type: Bug > Components: authentication >Affects Versions: 1.10.7 >Reporter: Rajender Reddy Kompally >Priority: Blocker > Attachments: airflow_rbac_issue.png > > > When we build airflow 1.10.7 from source and run using rbac = True option UI > is stuck and an't go to any view tabs. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6616) Add Tableau Hook and Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6616. --- Resolution: Duplicate [~rokm] What do you think? I think this issue is fixed now since AIRFLOW-6790 is merged into master and will be released in 2.0. If you think there is some functionality missing, please open up a new ticket specifying the exact Operator you are missing. Thanks :) Best Regards, Felix > Add Tableau Hook and Operators > -- > > Key: AIRFLOW-6616 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6616 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 2.0.0 >Reporter: Rok Mihevc >Priority: Minor > Fix For: 2.0.0 > > > Tableau (tableau.com) provides a python client > (github.com/tableau/server-client-python) that enables automation of certain > tasks (e.g. report sending). > Addition of Tableau hook and operators will enable using Tableau to generate > and send as steps in a wider ETL process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6861) Remove pool_full on TaskInstance
[ https://issues.apache.org/jira/browse/AIRFLOW-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6861. --- Fix Version/s: 1.10.10 Assignee: Kamil Bregula Resolution: Fixed > Remove pool_full on TaskInstance > > > Key: AIRFLOW-6861 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6861 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-6790) Add basic Tableau Integration
[ https://issues.apache.org/jira/browse/AIRFLOW-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6790 started by Felix Uellendall. - > Add basic Tableau Integration > - > > Key: AIRFLOW-6790 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6790 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 2.0.0, 1.10.9 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > Adds a basic Tableau Integration which consists of: > * TableauHook - wraps the tableauserverclient python library in an > AirflowHook to connect to a Tableau Server > * TableauRefreshWorkbookOperator - refreshes a Tableau workbook / extract > * TableauJobStatusSensor - pokes the status of a Tableau Job until it has > finished -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6790) Add basic Tableau Integration
Felix Uellendall created AIRFLOW-6790: - Summary: Add basic Tableau Integration Key: AIRFLOW-6790 URL: https://issues.apache.org/jira/browse/AIRFLOW-6790 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Affects Versions: 1.10.9, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall Adds a basic Tableau Integration which consists of: * TableauHook - wraps the tableauserverclient python library in an AirflowHook to connect to a Tableau Server * TableauRefreshWorkbookOperator - refreshes a Tableau workbook / extract * TableauJobStatusSensor - pokes the status of a Tableau Job until it has finished -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6751) Pin Werkzeug < 1.0.0
[ https://issues.apache.org/jira/browse/AIRFLOW-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-6751: -- Summary: Pin Werkzeug < 1.0.0 (was: Werkzeug 1.0 release breaks airflow) > Pin Werkzeug < 1.0.0 > > > Key: AIRFLOW-6751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6751 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.9 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6751) Werkzeug 1.0 release breaks airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032282#comment-17032282 ] Felix Uellendall edited comment on AIRFLOW-6751 at 2/7/20 10:30 AM: [~potiuk] 1.10.8 has already been released. So I set it to 1.10.9 and for your information there were already a ticket 6750 that describes the same issue. was (Author: feluelle): [~potiuk] 1.10.8 has already been released. So I set it to 1.10.9 > Werkzeug 1.0 release breaks airflow > --- > > Key: AIRFLOW-6751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6751 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.9 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6751) Werkzeug 1.0 release breaks airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-6751: -- Fix Version/s: (was: 1.10.8) 1.10.9 > Werkzeug 1.0 release breaks airflow > --- > > Key: AIRFLOW-6751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6751 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.9 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6751) Werkzeug 1.0 release breaks airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032282#comment-17032282 ] Felix Uellendall commented on AIRFLOW-6751: --- [~potiuk] 1.10.8 has already been released. So I set it to 1.10.9 > Werkzeug 1.0 release breaks airflow > --- > > Key: AIRFLOW-6751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6751 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.9 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5816) Add S3 to Snowflake operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5816. --- Resolution: Fixed > Add S3 to Snowflake operator > > > Key: AIRFLOW-5816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5816 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, operators >Affects Versions: 1.10.5 >Reporter: MOHAMMD SHAKEEL SHAIK >Assignee: MOHAMMD SHAKEEL SHAIK >Priority: Major > Fix For: 2.0.0 > > > We'd like a reusable way to LOAD data from a file stored in S3 into a > Snowflake table. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5816) Add S3 to Snowflake operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5816: -- Summary: Add S3 to Snowflake operator (was: S3 to Snowflake operator) > Add S3 to Snowflake operator > > > Key: AIRFLOW-5816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5816 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, operators >Affects Versions: 1.10.5 >Reporter: MOHAMMD SHAKEEL SHAIK >Assignee: MOHAMMD SHAKEEL SHAIK >Priority: Major > Fix For: 2.0.0 > > > We'd like a reusable way to LOAD data from a file stored in S3 into a > Snowflake table. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6245) Allow custom waiters for AWS batch jobs
[ https://issues.apache.org/jira/browse/AIRFLOW-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6245. --- Resolution: Fixed > Allow custom waiters for AWS batch jobs > --- > > Key: AIRFLOW-6245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6245 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.6 >Reporter: Darren Weber >Assignee: Darren Weber >Priority: Minor > Labels: AWS, aws-batch > Fix For: 2.0.0, 1.10.8 > > > The botocore waiter for AWS batch jobs has not been merged and released for > several years, i.e. > - [https://github.com/boto/botocore/pull/1307] > While working on this Airflow issue, I've also pushed up a PR on botocore to > use a default waiter with exponential backoff and add an option to use a > custom function for the delays between polling status, see > - [https://github.com/boto/botocore/issues/1915] > - [https://github.com/boto/botocore/pull/1921] > > For Airflow, adopt something from botocore PR-1307 as an example to create a > default batch job waiter in Airflow. As a guide to creating a custom waiter, > see > - [https://www.2ndwatch.com/blog/use-waiters-boto3-write/] > - > [https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5931) Spawning new python interpreter for every task slow
[
https://issues.apache.org/jira/browse/AIRFLOW-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992434#comment-16992434
]
Felix Uellendall commented on AIRFLOW-5931:
---
[~ash] don't forget to resolve the issue :) 1.10.7?
> Spawning new python interpreter for every task slow
> ---
>
> Key: AIRFLOW-5931
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors, worker
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Major
>
> There are a number of places in the Executors and Task Runners where we spawn
> a whole new python interpreter.
> My profiling has shown that this is slow. Rather than running a fresh python
> interpreter which then has to re-load all of Airflow and its dependencies we
> should use {{os.fork}} when it is available/suitable which should speed up
> task running, espeically for short lived tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5583) Extend the "DAG Details" page to display the start_date / end_date
[ https://issues.apache.org/jira/browse/AIRFLOW-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5583. --- Fix Version/s: 1.10.7 Resolution: Fixed > Extend the "DAG Details" page to display the start_date / end_date > -- > > Key: AIRFLOW-5583 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5583 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.5 >Reporter: Giovanni >Assignee: Giovanni >Priority: Major > Fix For: 1.10.7 > > Attachments: Screenshot from 2019-10-02 11-37-01.jpg > > > Extend the "DAG Details" page to display the start_date / end_date. > Attached screnshoot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3722) Improve BiqQueryHook test coverage
[ https://issues.apache.org/jira/browse/AIRFLOW-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982923#comment-16982923 ] Felix Uellendall commented on AIRFLOW-3722: --- No, don't think so. At least not in the near future. > Improve BiqQueryHook test coverage > -- > > Key: AIRFLOW-3722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3722 > Project: Apache Airflow > Issue Type: Test > Components: gcp >Reporter: Felix Uellendall >Priority: Major > > There are currently many lines not being tested. > > This Ticket will improve the overall test coverage of this Hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3913) Search Button overrides existing GET Parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982649#comment-16982649 ] Felix Uellendall commented on AIRFLOW-3913: --- You are right. Thanks. > Search Button overrides existing GET Parameter > -- > > Key: AIRFLOW-3913 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3913 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Priority: Minor > Labels: GoodForNewContributors, beginner, starter > > When you search for DAGs the url will be reset to the base_url + > "?search=". > So if you have hided paused dags for example and then search for something > you also get paused dags. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3912) Hide/Show Paused DAGs Button overrides existing GET Parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall closed AIRFLOW-3912. - Resolution: Fixed > Hide/Show Paused DAGs Button overrides existing GET Parameter > - > > Key: AIRFLOW-3912 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3912 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Assignee: Alejandro >Priority: Minor > Labels: GoodForNewContributors, beginner, starter > > When you click on the "Hide Paused DAGs" / "Show Paused DAGs" Button the url > will be reset to the base_url + "?show_paused=True" / "?show_paused=False". > This can be bad when you want to search for dags for example and then show > only the paused dags. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3912) Hide/Show Paused DAGs Button overrides existing GET Parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982650#comment-16982650 ] Felix Uellendall commented on AIRFLOW-3912: --- You are right. Thanks. > Hide/Show Paused DAGs Button overrides existing GET Parameter > - > > Key: AIRFLOW-3912 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3912 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Assignee: Alejandro >Priority: Minor > Labels: GoodForNewContributors, beginner, starter > > When you click on the "Hide Paused DAGs" / "Show Paused DAGs" Button the url > will be reset to the base_url + "?show_paused=True" / "?show_paused=False". > This can be bad when you want to search for dags for example and then show > only the paused dags. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3913) Search Button overrides existing GET Parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall closed AIRFLOW-3913. - Resolution: Fixed > Search Button overrides existing GET Parameter > -- > > Key: AIRFLOW-3913 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3913 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Priority: Minor > Labels: GoodForNewContributors, beginner, starter > > When you search for DAGs the url will be reset to the base_url + > "?search=". > So if you have hided paused dags for example and then search for something > you also get paused dags. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-3682) Default aws_conn_id is wrong for emr_base_sensor and emr_create_job_flow_operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-3682. --- Resolution: Fixed > Default aws_conn_id is wrong for emr_base_sensor and > emr_create_job_flow_operator > - > > Key: AIRFLOW-3682 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3682 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.1 >Reporter: Jonas Laake >Assignee: Jonas Laake >Priority: Minor > Fix For: 2.0.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Change default aws_conn_id in emr_terminate_job_flow_operator, > emr_create_job_flow_operator and emr_add_steps_operator for consistency with > emr_base_sensor > In the emr_base_sensor, the default aws_conn_id is set to 'aws_default'. It > should be the same for the emr_add_steps_operator, > emr_terminate_job_flow_operator and the emr_create_job_flow_operator, so they > land inside the same boto3 session. > For the current implementation it leads to errors where the EMR cluster is > not found by the emr_step_sensor. > Solution is to change aws_conn_id in both files to 'aws_default'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6063) Remove astroid dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-6063: -- Description: Note this dependency was only meant to temporarily fix an issue with pylint and astroid. See https://github.com/PyCQA/pylint/issues/3123 for more information. > Remove astroid dependency > - > > Key: AIRFLOW-6063 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6063 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > Note this dependency was only meant to temporarily fix an issue with pylint > and astroid. > See https://github.com/PyCQA/pylint/issues/3123 for more information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6063) Remove astroid dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-6063: -- Summary: Remove astroid dependency (was: Unpin pylint version) > Remove astroid dependency > - > > Key: AIRFLOW-6063 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6063 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6063) Unpin pylint version
Felix Uellendall created AIRFLOW-6063: - Summary: Unpin pylint version Key: AIRFLOW-6063 URL: https://issues.apache.org/jira/browse/AIRFLOW-6063 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5313) Add params support for awsbatch_operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5313. --- Fix Version/s: 1.10.7 Resolution: Fixed > Add params support for awsbatch_operator > > > Key: AIRFLOW-5313 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5313 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.4 >Reporter: Jonathan R Owen >Assignee: Jonathan R Owen >Priority: Minor > Fix For: 1.10.7 > > > AWS Batch supports [passing parameters to > jobs|https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job]. > Update AWSBatchOperator to include an optional parameters value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5582. --- Resolution: Fixed > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > Fix For: 1.10.7 > > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5582: -- Fix Version/s: 1.10.7 > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > Fix For: 1.10.7 > > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4051) MySQL hook bulk_load function doesn't work by default with MySql 8
[
https://issues.apache.org/jira/browse/AIRFLOW-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974990#comment-16974990
]
Felix Uellendall commented on AIRFLOW-4051:
---
[~ash] The mysql client does log a meaningful message. [~jackjack10] tested it
https://github.com/apache/airflow/pull/6575#issuecomment-554027473
In my opinion this issue is "solved" or can be closed.
> MySQL hook bulk_load function doesn't work by default with MySql 8
> --
>
> Key: AIRFLOW-4051
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4051
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: mysql
>
> This function executes {{LOAD DATA LOCAL ... }}which is disabled by default
> in MySQL 8 due to security concerns:
> https://dev.mysql.com/doc/refman/8.0/en/load-data-local.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-2999) Add S3DownloadOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974097#comment-16974097 ] Felix Uellendall edited comment on AIRFLOW-2999 at 11/14/19 9:50 AM: - If I would split this into "S3DownloadOperator" >> "MySqlUploadOperator" I would have a state in-between tasks that I think should not be. was (Author: feluelle): If I would split this into "S3DownloadOperator" >> "MySqlUploadOperator" I would have a state in between tasks that I think should not be. > Add S3DownloadOperator > -- > > Key: AIRFLOW-2999 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2999 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > > The [S3_hook > |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177] > has get_key method that returns boto3.s3.Object it also has load_file method > which loads file from local file system to S3. > > What it doesn't have is a method to download a file from S3 to the local file > system. > Basicly it should be something very simple... an extention to the get_key > method with parameter to the destination on local file system adding a code > for taking the boto3.s3.Object and save it on the disk. Note: that it can be > more than 1 file if the user choose a folder in S3. > > +*Update:*+ > As discussed in comments instead having the property in the hook it's better > to mirror the GoogleCloudStorageDownloadOperator and have S3DownloadOperator > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2999) Add S3DownloadOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974097#comment-16974097 ] Felix Uellendall commented on AIRFLOW-2999: --- If I would split this into "S3DownloadOperator" >> "MySqlUploadOperator" I would have a state in between tasks that I think should not be. > Add S3DownloadOperator > -- > > Key: AIRFLOW-2999 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2999 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > > The [S3_hook > |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177] > has get_key method that returns boto3.s3.Object it also has load_file method > which loads file from local file system to S3. > > What it doesn't have is a method to download a file from S3 to the local file > system. > Basicly it should be something very simple... an extention to the get_key > method with parameter to the destination on local file system adding a code > for taking the boto3.s3.Object and save it on the disk. Note: that it can be > more than 1 file if the user choose a folder in S3. > > +*Update:*+ > As discussed in comments instead having the property in the hook it's better > to mirror the GoogleCloudStorageDownloadOperator and have S3DownloadOperator > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2999) Add S3DownloadOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974094#comment-16974094 ] Felix Uellendall commented on AIRFLOW-2999: --- In general if you think of the download as a "transfer" operation (where it really matters where the file will be downloaded to) I agree with it being an Operator. But I want to use as a Hook functionality so I can temporarily download it to a local machine and upload it from there to database like mysql. See https://issues.apache.org/jira/browse/AIRFLOW-5907 (+ PR) > Add S3DownloadOperator > -- > > Key: AIRFLOW-2999 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2999 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > > The [S3_hook > |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177] > has get_key method that returns boto3.s3.Object it also has load_file method > which loads file from local file system to S3. > > What it doesn't have is a method to download a file from S3 to the local file > system. > Basicly it should be something very simple... an extention to the get_key > method with parameter to the destination on local file system adding a code > for taking the boto3.s3.Object and save it on the disk. Note: that it can be > more than 1 file if the user choose a folder in S3. > > +*Update:*+ > As discussed in comments instead having the property in the hook it's better > to mirror the GoogleCloudStorageDownloadOperator and have S3DownloadOperator > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5924: -- Fix Version/s: 2.0.0 > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5921 started by Felix Uellendall. - > Add bulk_load_custom to MySqlHook > - > > Key: AIRFLOW-5921 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5921 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5924 started by Felix Uellendall. - > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5922 started by Felix Uellendall. - > Add option to specify the mysql client library used in MySqlHook > > > Key: AIRFLOW-5922 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5922 > Project: Apache Airflow > Issue Type: New Feature > Components: dependencies, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5908) Add download_file to S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5908 started by Felix Uellendall. - > Add download_file to S3 Hook > > > Key: AIRFLOW-5908 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5907) Add S3 to MySql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5907 started by Felix Uellendall. - > Add S3 to MySql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) Add S3 to MySql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to MySql Operator (was: Add S3 to Mysql Operator) > Add S3 to MySql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) Add S3 to Mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to Mysql Operator (was: S3 to Mysql Operator) > Add S3 to Mysql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
Felix Uellendall created AIRFLOW-5924: - Summary: Automatically unify bucket name and key in S3Hook Key: AIRFLOW-5924 URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 Project: Apache Airflow Issue Type: New Feature Components: aws, hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
Felix Uellendall created AIRFLOW-5922: - Summary: Add option to specify the mysql client library used in MySqlHook Key: AIRFLOW-5922 URL: https://issues.apache.org/jira/browse/AIRFLOW-5922 Project: Apache Airflow Issue Type: New Feature Components: dependencies, hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) S3 to Mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Affects Version/s: 2.0.0 > S3 to Mysql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5908) Add download_file to S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5908: -- Affects Version/s: 2.0.0 > Add download_file to S3 Hook > > > Key: AIRFLOW-5908 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
Felix Uellendall created AIRFLOW-5921: - Summary: Add bulk_load_custom to MySqlHook Key: AIRFLOW-5921 URL: https://issues.apache.org/jira/browse/AIRFLOW-5921 Project: Apache Airflow Issue Type: New Feature Components: hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5442) Druid broker hook get pandas DataFrame implementation
[ https://issues.apache.org/jira/browse/AIRFLOW-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5442: -- Fix Version/s: (was: 1.10.6) 1.10.7 > Druid broker hook get pandas DataFrame implementation > - > > Key: AIRFLOW-5442 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5442 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.5 >Reporter: Sayed Mohammad Hossein Torabi >Assignee: Sayed Mohammad Hossein Torabi >Priority: Minor > Fix For: 1.10.7 > > > The *get_pandas_df* of *DruidDbApiHook* returns NotImplementedError. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5908) Add download_file to S3 Hook
Felix Uellendall created AIRFLOW-5908: - Summary: Add download_file to S3 Hook Key: AIRFLOW-5908 URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 Project: Apache Airflow Issue Type: New Feature Components: aws, hooks Affects Versions: 1.10.6 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5907) S3 to Mysql Operator
Felix Uellendall created AIRFLOW-5907: - Summary: S3 to Mysql Operator Key: AIRFLOW-5907 URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 Project: Apache Airflow Issue Type: New Feature Components: operators Affects Versions: 1.10.6 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-3031) S3 to Snowflake Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972324#comment-16972324 ] Felix Uellendall edited comment on AIRFLOW-3031 at 11/12/19 12:17 PM: -- [~bhn021] are you still working on this? Because unfortunately there is another one https://issues.apache.org/jira/browse/AIRFLOW-5816 with a PR. was (Author: feluelle): [~bhn021] are you still working on this? > S3 to Snowflake Operator > > > Key: AIRFLOW-3031 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3031 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Brian Nguyen >Assignee: Brian Nguyen >Priority: Minor > Labels: operators > Original Estimate: 24h > Remaining Estimate: 24h > > Create an operator to load csv or json files from S3 into snowflake, similar > to the operator created for s3_to_redshift. > > I've already written most of it and just need to write the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3031) S3 to Snowflake Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972324#comment-16972324 ] Felix Uellendall commented on AIRFLOW-3031: --- [~bhn021] are you still working on this? > S3 to Snowflake Operator > > > Key: AIRFLOW-3031 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3031 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Brian Nguyen >Assignee: Brian Nguyen >Priority: Minor > Labels: operators > Original Estimate: 24h > Remaining Estimate: 24h > > Create an operator to load csv or json files from S3 into snowflake, similar > to the operator created for s3_to_redshift. > > I've already written most of it and just need to write the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5883: -- Fix Version/s: 1.10.7 > Fix inefficient sqlalchemy .count() queries > --- > > Key: AIRFLOW-5883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5883 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.6 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 1.10.7 > > > SQLAlchemy .count() uses a subquery and is inefficient -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5883. --- Resolution: Fixed > Fix inefficient sqlalchemy .count() queries > --- > > Key: AIRFLOW-5883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5883 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.6 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 1.10.7 > > > SQLAlchemy .count() uses a subquery and is inefficient -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5874) Use poke_interval of 0 to make EMR tests run green
[ https://issues.apache.org/jira/browse/AIRFLOW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5874. --- Resolution: Fixed > Use poke_interval of 0 to make EMR tests run green > -- > > Key: AIRFLOW-5874 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5874 > Project: Apache Airflow > Issue Type: Sub-task > Components: operators >Affects Versions: 1.10.6 >Reporter: Felix Uellendall >Priority: Trivial > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5874) Use poke_interval of 0 to make EMR tests run green
Felix Uellendall created AIRFLOW-5874: - Summary: Use poke_interval of 0 to make EMR tests run green Key: AIRFLOW-5874 URL: https://issues.apache.org/jira/browse/AIRFLOW-5874 Project: Apache Airflow Issue Type: Sub-task Components: operators Affects Versions: 1.10.6 Reporter: Felix Uellendall Fix For: 1.10.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5743) Move Gogle PubSub to providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-5743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5743. --- Fix Version/s: 2.0.0 Resolution: Done > Move Gogle PubSub to providers package > -- > > Key: AIRFLOW-5743 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5743 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Rich Dean >Priority: Minor > Fix For: 2.0.0 > > > Move: > airflow.gcp.operators.pubsub -> > airflow.providers.google.cloud.operators.pubsub > airflow.gcp.hooks.pubsub -> airflow.providers.google.cloud.hooks.pubsub > and also relevant tests and example DAGs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5758) Support the custom cursor classes for the PostgreSQL hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5758. --- Resolution: Done > Support the custom cursor classes for the PostgreSQL hook > - > > Key: AIRFLOW-5758 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5758 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.7 >Reporter: MinJae Kwon >Assignee: MinJae Kwon >Priority: Minor > Fix For: 1.10.7 > > > MySQL hook supports the custom cursor classes currently, but PostgreSQL hook > does not. > > It will be good that PostgreSQL also supports the custom classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5837) Remove 'azure-storage-blob<12.0' dependency
Felix Uellendall created AIRFLOW-5837: - Summary: Remove 'azure-storage-blob<12.0' dependency Key: AIRFLOW-5837 URL: https://issues.apache.org/jira/browse/AIRFLOW-5837 Project: Apache Airflow Issue Type: Improvement Components: dependencies Affects Versions: 2.0.0, 1.10.6 Reporter: Felix Uellendall Remove this dependency after https://github.com/snowflakedb/snowflake-connector-python/issues/225 is solved and released. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5683) Add propagate_skipped_state to SubDagOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5683: -- Summary: Add propagate_skipped_state to SubDagOperator (was: Add SubDagOperator that propagates skipped state) > Add propagate_skipped_state to SubDagOperator > - > > Key: AIRFLOW-5683 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5683 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5683) Add propagate_skipped_state to SubDagOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5683: -- Description: Currently there is no way of telling the parent dag of a sub dag that an essential task has been skipped. This PR addresses this issue by adding a new propagate_skipped_state option to the SubDagOperator. Background story: https://lists.apache.org/thread.html/0eefd459a502c5100d792416f8ba720302aa49a8906fe6ea4ec8fca4@%3Cdev.airflow.apache.org%3E > Add propagate_skipped_state to SubDagOperator > - > > Key: AIRFLOW-5683 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5683 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > Currently there is no way of telling the parent dag of a sub dag that an > essential task has been skipped. This PR addresses this issue by adding a new > propagate_skipped_state option to the SubDagOperator. > Background story: > https://lists.apache.org/thread.html/0eefd459a502c5100d792416f8ba720302aa49a8906fe6ea4ec8fca4@%3Cdev.airflow.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5798) Set default to None for external_task_id in ExternalTaskSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5798. --- Resolution: Done > Set default to None for external_task_id in ExternalTaskSensor > -- > > Key: AIRFLOW-5798 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5798 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5798) Set default to None for external_task_id in ExternalTaskSensor
Felix Uellendall created AIRFLOW-5798: - Summary: Set default to None for external_task_id in ExternalTaskSensor Key: AIRFLOW-5798 URL: https://issues.apache.org/jira/browse/AIRFLOW-5798 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.5 Reporter: Felix Uellendall Fix For: 1.10.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5766) example_http_operator is failing due to 404 errors
[ https://issues.apache.org/jira/browse/AIRFLOW-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5766. --- Resolution: Done > example_http_operator is failing due to 404 errors > -- > > Key: AIRFLOW-5766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5766 > Project: Apache Airflow > Issue Type: Improvement > Components: examples >Affects Versions: 1.10.7 >Reporter: MinJae Kwon >Assignee: MinJae Kwon >Priority: Minor > Fix For: 1.10.7 > > > example_http_operator is using `https://google.com` as base URL, and does > each HTTP request to `/api/v1.0/node` endpoint. But > `https://google.com/api/v1.0/node` returns 404 error, so this example always > fails. > > So it should use another working website or URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5742) Move Cloud Vision to providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5742. --- Fix Version/s: 2.0.0 Resolution: Done > Move Cloud Vision to providers package > -- > > Key: AIRFLOW-5742 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5742 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Louis Guitton >Priority: Minor > Fix For: 2.0.0 > > > Move: > airflow.gcp.operators.vision -> > airflow.providers.google.cloud.operators.vision > airflow.gcp.hooks.vision -> airflow.providers.google.cloud.hooks.vision > and also relevant tests and example DAGs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5749) Support the "blocks" component for slack operators
[ https://issues.apache.org/jira/browse/AIRFLOW-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5749. --- Fix Version/s: 1.10.7 Resolution: Done > Support the "blocks" component for slack operators > -- > > Key: AIRFLOW-5749 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5749 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.6 >Reporter: MinJae Kwon >Assignee: MinJae Kwon >Priority: Minor > Fix For: 1.10.7 > > > We should support "blocks" component for the Slack operators as for the Slack > webhookhook (Related: https://issues.apache.org/jira/browse/AIRFLOW-5693) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4162) Support layout blocks in Slack operators
[ https://issues.apache.org/jira/browse/AIRFLOW-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-4162. --- Fix Version/s: 1.10.7 Resolution: Done > Support layout blocks in Slack operators > > > Key: AIRFLOW-4162 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4162 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.2 >Reporter: Bumsoo Kim >Assignee: Bumsoo Kim >Priority: Minor > Labels: features > Fix For: 1.10.7 > > > Slack introduced a new API called "*layout blocks"* to create rich message > layouts. > [https://api.slack.com/messaging/composing/layouts] > > We can utilize this new feature by simply adding `blocks` array field to the > payload. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5690) Task heartbeat is being reported as a warning
[
https://issues.apache.org/jira/browse/AIRFLOW-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall resolved AIRFLOW-5690.
---
Fix Version/s: 1.10.7
Resolution: Fixed
> Task heartbeat is being reported as a warning
> -
>
> Key: AIRFLOW-5690
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5690
> Project: Apache Airflow
> Issue Type: Bug
> Components: worker
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Aditya Vishwakarma
>Priority: Minor
> Fix For: 1.10.7
>
>
> This bug is on the master branch and v1-10-stable.
> All my workers have logs like this occurring every {{JOB_HEARTBEAT_SEC}}
> seconds. It is overloading my logging system.
> {{local_task_job.py:124}} WARNING [1 - MainThread] - Time since last
> heartbeat(0.02 s) < heartrate(10.0 s), sleeping for 9.983787 s
> I think this is a case of mistaken log level. It should probably be a debug,
> or even a trace log level.
> [https://github.com/adityav/airflow/blob/master/airflow/jobs/local_task_job.py#L118]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5693) Support the "blocks" component for the Slack messages
[ https://issues.apache.org/jira/browse/AIRFLOW-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5693. --- Resolution: Done > Support the "blocks" component for the Slack messages > - > > Key: AIRFLOW-5693 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5693 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.6 >Reporter: MinJae Kwon >Assignee: MinJae Kwon >Priority: Minor > Fix For: 1.10.7 > > > Currently, the slack webhook hook does not support "blocks" component for the > Slack messages.([https://api.slack.com/reference/block-kit/blocks]) > > So, we should support the "blocks" field to slack webhook hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5683) Add SubDagOperator that propagates skipped state
[ https://issues.apache.org/jira/browse/AIRFLOW-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5683 started by Felix Uellendall. - > Add SubDagOperator that propagates skipped state > > > Key: AIRFLOW-5683 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5683 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5683) Add SubDagOperator that propagates skipped state
Felix Uellendall created AIRFLOW-5683: - Summary: Add SubDagOperator that propagates skipped state Key: AIRFLOW-5683 URL: https://issues.apache.org/jira/browse/AIRFLOW-5683 Project: Apache Airflow Issue Type: New Feature Components: operators Affects Versions: 1.10.6 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer
[ https://issues.apache.org/jira/browse/AIRFLOW-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953002#comment-16953002 ] Felix Uellendall commented on AIRFLOW-5643: --- [~ash] I set Fix Version to 2.0 since you said you want to cut the rc tomorrow. I suggest we add it to 1.10.6 if we have a rc2 :D > S3Hook logic is duplicated and could be clearer > --- > > Key: AIRFLOW-5643 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5643 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, hooks >Affects Versions: 1.10.5 >Reporter: Louis Guitton >Assignee: Louis Guitton >Priority: Trivial > Fix For: 2.0.0 > > > S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj > [https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539] > Instead, we should stay consistent : S3Hook.load_string is already delegating > the logic to S3Hook.load_bytes, so we can use the same approach to delegate > to S3Hook.load_file_obj -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer
[ https://issues.apache.org/jira/browse/AIRFLOW-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5643. --- Resolution: Fixed > S3Hook logic is duplicated and could be clearer > --- > > Key: AIRFLOW-5643 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5643 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, hooks >Affects Versions: 1.10.5 >Reporter: Louis Guitton >Assignee: Louis Guitton >Priority: Trivial > Fix For: 2.0.0 > > > S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj > [https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539] > Instead, we should stay consistent : S3Hook.load_string is already delegating > the logic to S3Hook.load_bytes, so we can use the same approach to delegate > to S3Hook.load_file_obj -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer
[ https://issues.apache.org/jira/browse/AIRFLOW-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5643: -- Fix Version/s: 2.0.0 > S3Hook logic is duplicated and could be clearer > --- > > Key: AIRFLOW-5643 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5643 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, hooks >Affects Versions: 1.10.5 >Reporter: Louis Guitton >Assignee: Louis Guitton >Priority: Trivial > Fix For: 2.0.0 > > > S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj > [https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539] > Instead, we should stay consistent : S3Hook.load_string is already delegating > the logic to S3Hook.load_bytes, so we can use the same approach to delegate > to S3Hook.load_file_obj -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-4661) Make airflow/config_templates Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-4661: -- Fix Version/s: 2.0.0 > Make airflow/config_templates Pylint compatible > --- > > Key: AIRFLOW-4661 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4661 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Arkadiusz Bach >Priority: Major > Fix For: 2.0.0 > > > Fix all Pylint messages in airflow/config_templates. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4661) Make airflow/config_templates Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-4661. --- Resolution: Fixed > Make airflow/config_templates Pylint compatible > --- > > Key: AIRFLOW-4661 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4661 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Arkadiusz Bach >Priority: Major > > Fix all Pylint messages in airflow/config_templates. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Component/s: operators > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.5 >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5395) Fix unloading with column names containing a dot in RedshiftToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5395 started by Felix Uellendall. - > Fix unloading with column names containing a dot in RedshiftToS3Operator > > > Key: AIRFLOW-5395 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5395 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Affects Version/s: 1.10.5 > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.5 >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Summary: Switch to HEADER arg to unload data from Redshift to S3 (was: Fix header rendering in the redshift-to-s3 plugin) > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-3783) Fix header rendering in the redshift-to-s3 plugin
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall reassigned AIRFLOW-3783: - Assignee: Felix Uellendall (was: Shreya Chakraborty) > Fix header rendering in the redshift-to-s3 plugin > - > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5572) Clear task reschedules when clearing task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5572. --- Fix Version/s: 1.10.6 Resolution: Fixed > Clear task reschedules when clearing task instances > --- > > Key: AIRFLOW-5572 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5572 > Project: Apache Airflow > Issue Type: Bug > Components: core, models >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5387) Pagination is broken when the config variable hide_paused_dags_by_default is set to True because showPaused variable is removed
[
https://issues.apache.org/jira/browse/AIRFLOW-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-5387:
--
Fix Version/s: 1.10.6
> Pagination is broken when the config variable hide_paused_dags_by_default is
> set to True because showPaused variable is removed
> ---
>
> Key: AIRFLOW-5387
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5387
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4
>Reporter: Guillem Lefait
>Assignee: Alejandro
>Priority: Minor
> Fix For: 1.10.6
>
>
> Url parameters are clean to remove useless parameters:
> {code:python}
> // https://github.com/apache/airflow/blob/master/airflow/www/utils.py#L68-L71
> if 'showPaused' in kwargs:
> v = kwargs['showPaused']
> if v or v is None:
> kwargs.pop('showPaused')
> {code}
> When the configuration variable _hide_paused_dags_by_default_ is set to True,
> removing the parameter showPaused disable access to the correct paginated
> pages when navigating to "Show Paused DAGs" .
> Pagination is fine for the first page but when going on second page, only
> "active" DAGs are presented.
> Fix:
> - showPaused parameter should not be removed and therefore lines 68 to 71
> (included) could be removed.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5387) Pagination is broken when the config variable hide_paused_dags_by_default is set to True because showPaused variable is removed
[
https://issues.apache.org/jira/browse/AIRFLOW-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall resolved AIRFLOW-5387.
---
Resolution: Fixed
> Pagination is broken when the config variable hide_paused_dags_by_default is
> set to True because showPaused variable is removed
> ---
>
> Key: AIRFLOW-5387
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5387
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4
>Reporter: Guillem Lefait
>Assignee: Alejandro
>Priority: Minor
> Fix For: 1.10.6
>
>
> Url parameters are clean to remove useless parameters:
> {code:python}
> // https://github.com/apache/airflow/blob/master/airflow/www/utils.py#L68-L71
> if 'showPaused' in kwargs:
> v = kwargs['showPaused']
> if v or v is None:
> kwargs.pop('showPaused')
> {code}
> When the configuration variable _hide_paused_dags_by_default_ is set to True,
> removing the parameter showPaused disable access to the correct paginated
> pages when navigating to "Show Paused DAGs" .
> Pagination is fine for the first page but when going on second page, only
> "active" DAGs are presented.
> Fix:
> - showPaused parameter should not be removed and therefore lines 68 to 71
> (included) could be removed.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5572) [AIRFLOW-5572] Clear task reschedules when clearing task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5572: -- Summary: [AIRFLOW-5572] Clear task reschedules when clearing task instances (was: Clearing Task Instances with reschedule mode does not clear reschedules during up_for_reschedule) > [AIRFLOW-5572] Clear task reschedules when clearing task instances > -- > > Key: AIRFLOW-5572 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5572 > Project: Apache Airflow > Issue Type: Bug > Components: core, models >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5572) Clear task reschedules when clearing task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5572: -- Summary: Clear task reschedules when clearing task instances (was: [AIRFLOW-5572] Clear task reschedules when clearing task instances) > Clear task reschedules when clearing task instances > --- > > Key: AIRFLOW-5572 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5572 > Project: Apache Airflow > Issue Type: Bug > Components: core, models >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5572) Clearing Task Instances with reschedule mode does not clear reschedules during up_for_reschedule
Felix Uellendall created AIRFLOW-5572: - Summary: Clearing Task Instances with reschedule mode does not clear reschedules during up_for_reschedule Key: AIRFLOW-5572 URL: https://issues.apache.org/jira/browse/AIRFLOW-5572 Project: Apache Airflow Issue Type: Bug Components: core, models Affects Versions: 1.10.5 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5553) mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be encoded whenever a BQ schema is not explicitly specified
[
https://issues.apache.org/jira/browse/AIRFLOW-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-5553:
--
Fix Version/s: (was: 1.10.6)
2.0.0
> mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be
> encoded whenever a BQ schema is not explicitly specified
> ---
>
> Key: AIRFLOW-5553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5553
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Nikolay Stoynov
>Assignee: Nikolay Stoynov
>Priority: Minor
> Labels: easyfix
> Fix For: 2.0.0
>
>
>
> The MySqlGoogleCloudStorageOperator currently deals with `bytes` values.
> However, the way it detects when a value is of type `bytes` is dependent on
> the column type in the BQ schema provided.
> Therefore, whenever a BQ schema is not provided, the value is not dealt with
> and a serialization issue occurs:
> {noformat}
> [2018-10-11 17:59:36,056] INFO
> airflow.task.task_runner.bash_task_runner.BashTaskRunner _read_task_logs -
> Job 11906: Subtask my_subtask TypeError: Object of type 'bytes' is not JSON
> serializable{noformat}
> This should be a trivial fix but needs to be addressed so that the check for
> a bytes value is based on the type of the variable too.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
