[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172902#comment-17172902
]
Tomasz Urbaszek edited comment on AIRFLOW-5071 at 8/7/20, 5:37 AM:
---
[~antontimenko] I'm running 100 DAGs of the following type and no faliure
class TestSensor(BaseSensorOperator):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.mode = "RESCHEDULE"
def poke(self, context):
return choice([True, False, False])
args = \{"owner": "airflow", "start_date": days_ago(1)}
with DAG(
dag_id="%s",
is_paused_upon_creation=False,
max_active_runs=1,
default_args=args,
schedule_interval="* * * * *",
) as dag:
start = TestSensor(task_id="start")
end = TestSensor(task_id="end")
for i in range(100):
next = TestSensor(task_id=f"next_\{i}")
start >> next >> end
I'm using Airflow 1.10.6 on Composer
was (Author: turbaszek):
[~antontimenko] I'm running 100 DAGs of the following type:
class TestSensor(BaseSensorOperator):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.mode = "RESCHEDULE"
def poke(self, context):
return choice([True, False, False])
args = \{"owner": "airflow", "start_date": days_ago(1)}
with DAG(
dag_id="%s",
is_paused_upon_creation=False,
max_active_runs=1,
default_args=args,
schedule_interval="* * * * *",
) as dag:
start = TestSensor(task_id="start")
end = TestSensor(task_id="end")
for i in range(100):
next = TestSensor(task_id=f"next_\{i}")
start >> next >> end
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172902#comment-17172902
]
Tomasz Urbaszek commented on AIRFLOW-5071:
--
[~antontimenko] I'm running 100 DAGs of the following type:
class TestSensor(BaseSensorOperator):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.mode = "RESCHEDULE"
def poke(self, context):
return choice([True, False, False])
args = \{"owner": "airflow", "start_date": days_ago(1)}
with DAG(
dag_id="%s",
is_paused_upon_creation=False,
max_active_runs=1,
default_args=args,
schedule_interval="* * * * *",
) as dag:
start = TestSensor(task_id="start")
end = TestSensor(task_id="end")
for i in range(100):
next = TestSensor(task_id=f"next_\{i}")
start >> next >> end
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172148#comment-17172148
]
Tomasz Urbaszek edited comment on AIRFLOW-5071 at 8/6/20, 9:33 AM:
---
[~sgrzemski] the mismatch between db, executor and in-memory state is also my
guess.
[~sgrzemski] [~siberiancrane] Do you know how this can be easily replicated? I
was trying to replicate but with no result :<
And additonaly, if you are using CeleryExecutor then what broker do you use?
Rabbit / Redis? What backend as metadb? Is this managed solution or self-hosted?
was (Author: turbaszek):
[~sgrzemski] the mismatch between db, executor and in-memory state is also my
guess.
[~sgrzemski] [~siberiancrane] Do you know how this can be easily replicated? I
was trying to replicate but with no result :<
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172148#comment-17172148
]
Tomasz Urbaszek commented on AIRFLOW-5071:
--
[~sgrzemski] the mismatch between db, executor and in-memory state is also my
guess.
[~sgrzemski] [~siberiancrane] Do you know how this can be easily replicated? I
was trying to replicate but with no result :<
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164333#comment-17164333
]
Tomasz Urbaszek commented on AIRFLOW-5071:
--
I'm trying to replicate this issue but it's not an obvious one. [~msempere]
[~sgrzemski] [~ipeluffo] would you mind sharing your config setup? Max active
runs, dag concurrency, parallelism? Plus, are you using CeleryExecutor?
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163381#comment-17163381 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Hi [~stevepak] I addressed your questions in this PR: [https://github.com/apache/airflow/pull/9951] Btw. we migrated to Github issues, and in case of troubleshooting questions I recommend using our slack: [https://apache-airflow-slack.herokuapp.com|https://apache-airflow-slack.herokuapp.com/] > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162695#comment-17162695
]
Tomasz Urbaszek commented on AIRFLOW-5071:
--
[~sgrzemski] is `check_signals_table` a sensor?
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6685) Add ThresholdCheckOperator for Data Quality Checking
[
https://issues.apache.org/jira/browse/AIRFLOW-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-6685.
--
Fix Version/s: 2.0.0
Resolution: Done
> Add ThresholdCheckOperator for Data Quality Checking
> -
>
> Key: AIRFLOW-6685
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6685
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 2.0.0
>Reporter: alex l
>Assignee: alex l
>Priority: Major
> Fix For: 2.0.0
>
>
> This PR includes a new operator in *{{CheckOperator}}* that allows users to
> perform a threshold data quality check.
> *{{ThresholdCheckOperator}}* will check a single value, sql result against a
> threshold range, and will fail a task if it is outside this range. The lower
> and upper bound of the threshold can be defined as either a numeric values,
> or sql-statements that returns a numeric value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-848) Flower does not cleanup pid file on SIGTERM
[ https://issues.apache.org/jira/browse/AIRFLOW-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070715#comment-17070715 ] Tomasz Urbaszek commented on AIRFLOW-848: - [~dimberman] I did that only to webserver > Flower does not cleanup pid file on SIGTERM > --- > > Key: AIRFLOW-848 > URL: https://issues.apache.org/jira/browse/AIRFLOW-848 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.7.1.3 >Reporter: Rahul >Priority: Minor > > I am running a monit restart for celery flower. However, when I send a > SIGTERM signal to flower and start again, flower doesn't start up. This is > because of the pid file. I think it should clear this on sigterm/sigint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5801) CloudSqlProxyRunner always assumes GCP credentials are passed via file instead of JSON blob
[
https://issues.apache.org/jira/browse/AIRFLOW-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-5801.
--
Resolution: Done
> CloudSqlProxyRunner always assumes GCP credentials are passed via file
> instead of JSON blob
> ---
>
> Key: AIRFLOW-5801
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5801
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Dan Huang
>Assignee: Ephraim E Anierobi
>Priority: Major
> Labels: gsoc
> Fix For: 2.0.0
>
>
> See:
> *
> [https://github.com/apache/airflow/blob/fc4aa041b0f944d3b1f1b2d045a0518827f054a2/airflow/gcp/hooks/cloud_sql.py#L512]
> *
> [https://github.com/apache/airflow/blob/1.10.5/airflow/contrib/hooks/gcp_sql_hook.py#L488]
>
> Very quick fix here is to replace
>
> {code:java}
> if GCP_CREDENTIALS_KEY_PATH in connection.extra_dejson:
> {code}
> with
>
> {code:java}
> if connection.extra_dejson.get(GCP_CREDENTIALS_KEY_PATH):{code}
> And similarly for the lines below.
> But perhaps falsy values for connection keys should just be scrubbed?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6624) Improve webserver command with pidfile checking
[ https://issues.apache.org/jira/browse/AIRFLOW-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6624. -- Fix Version/s: 2.0.0 Resolution: Done > Improve webserver command with pidfile checking > --- > > Key: AIRFLOW-6624 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6624 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7099) Improve system test for cloud transfer service
[ https://issues.apache.org/jira/browse/AIRFLOW-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-7099. -- Fix Version/s: 2.0.0 Resolution: Done > Improve system test for cloud transfer service > -- > > Key: AIRFLOW-7099 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7099 > Project: Apache Airflow > Issue Type: Improvement > Components: examples, gcp, tests >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7100) Add GoogleAnalyticsGetAdsLinkOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-7100. -- Resolution: Done > Add GoogleAnalyticsGetAdsLinkOperator > - > > Key: AIRFLOW-7100 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7100 > Project: Apache Airflow > Issue Type: New Feature > Components: api >Affects Versions: 1.10.8 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7099) Improve system test for cloud transfer service
Tomasz Urbaszek created AIRFLOW-7099: Summary: Improve system test for cloud transfer service Key: AIRFLOW-7099 URL: https://issues.apache.org/jira/browse/AIRFLOW-7099 Project: Apache Airflow Issue Type: Improvement Components: examples, gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6978) Add PubSubPullOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6978. -- Fix Version/s: 2.0.0 Resolution: Done > Add PubSubPullOperator > -- > > Key: AIRFLOW-6978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6978 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Aleksander Nitecki >Priority: Major > Labels: operators, pull-request-available > Fix For: 2.0.0 > > > Add PubSubPullOperator - a non-blocking equivalent to Add PubSubPullSensor. > These two share most of functionality but ultimately serve different purpose > - while existing PubSubPullSensor will block the pipeline until it receives a > message, new PubSubPullOperator can be used to check for new messages > opportunistically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6732. -- Fix Version/s: 2.0.0 Resolution: Done > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7082) Remove catch_http_exception decorator in GCP hooks
[ https://issues.apache.org/jira/browse/AIRFLOW-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-7082. -- Fix Version/s: 2.0.0 Resolution: Done > Remove catch_http_exception decorator in GCP hooks > -- > > Key: AIRFLOW-7082 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7082 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7082) Remove catch_http_exception decorator in GCP hooks
Tomasz Urbaszek created AIRFLOW-7082: Summary: Remove catch_http_exception decorator in GCP hooks Key: AIRFLOW-7082 URL: https://issues.apache.org/jira/browse/AIRFLOW-7082 Project: Apache Airflow Issue Type: Improvement Components: gcp, hooks, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4972) Add Google Search Ads 360 integration
[ https://issues.apache.org/jira/browse/AIRFLOW-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek closed AIRFLOW-4972. Resolution: Fixed > Add Google Search Ads 360 integration > - > > Key: AIRFLOW-4972 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4972 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Labels: gsoc, gsoc2020, mentor > > Hi > This project lacks integration with the Google Search Ads 360 service. I > would be happy if Airflow had proper operators and hooks that integrate with > this service. > Product Documentation: https://developers.google.com/search-ads/ > API Documentation: > https://developers.google.com/resources/api-libraries/documentation/dfareporting/v3.3/python/latest/ > Lots of love -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4967) Add Cloud AutoML NL Entity Extraction integration
[ https://issues.apache.org/jira/browse/AIRFLOW-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-4967. -- Fix Version/s: 2.0.0 Resolution: Done > Add Cloud AutoML NL Entity Extraction integration > - > > Key: AIRFLOW-4967 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4967 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Hi > This project lacks integration with the Cloud AutoML NL Entity Extraction > service. I would be happy if Airflow had proper operators and hooks that > integrate with this service. > Product Documentation: > https://cloud.google.com/natural-language/automl/entity-analysis/docs/ > API Documentation: > https://googleapis.github.io/google-cloud-python/latest/automl/index.html > Love -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4969) Add Cloud AutoML Tables integration
[ https://issues.apache.org/jira/browse/AIRFLOW-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-4969. -- Fix Version/s: 2.0.0 Resolution: Done > Add Cloud AutoML Tables integration > --- > > Key: AIRFLOW-4969 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4969 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Hi > This project lacks integration with the Cloud AutoML Tables service. I would > be happy if Airflow had proper operators and hooks that integrate with this > service. > Product Documentation: https://cloud.google.com/automl-tables/docs/ > API Documentation: > https://googleapis.github.io/google-cloud-python/latest/automl/index.html > Love -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4968) Add Cloud AutoML NL Sentiment integration
[ https://issues.apache.org/jira/browse/AIRFLOW-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-4968. -- Fix Version/s: 2.0.0 Resolution: Done > Add Cloud AutoML NL Sentiment integration > - > > Key: AIRFLOW-4968 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4968 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Hi > This project lacks integration with the Cloud AutoML NL Sentiment service. I > would be happy if Airflow had proper operators and hooks that integrate with > this service. > Product Documentation: > https://cloud.google.com/natural-language/automl/sentiment/docs/ > API Documentation: > https://googleapis.github.io/google-cloud-python/latest/automl/index.html > Lots of love -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4966) Add Cloud AutoML NL Classification integration
[ https://issues.apache.org/jira/browse/AIRFLOW-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-4966. -- Fix Version/s: 2.0.0 Resolution: Done > Add Cloud AutoML NL Classification integration > -- > > Key: AIRFLOW-4966 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4966 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Hi > This project lacks integration with the Cloud AutoML NL Classification > service. I would be happy if Airflow had proper operators and hooks that > integrate with this service. > Product Documentation: https://cloud.google.com/natural-language/automl/docs/ > API Documentation: > https://googleapis.github.io/google-cloud-python/latest/automl/index.html > Love -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7075) Operators for storing information from GCS into Google Analytics
[ https://issues.apache.org/jira/browse/AIRFLOW-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-7075: - Summary: Operators for storing information from GCS into Google Analytics (was: Operators for storing ifnromation from GCS into Google Analytics) > Operators for storing information from GCS into Google Analytics > > > Key: AIRFLOW-7075 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7075 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7075) Operators for storing ifnromation from GCS into Google Analytics
Tomasz Urbaszek created AIRFLOW-7075: Summary: Operators for storing ifnromation from GCS into Google Analytics Key: AIRFLOW-7075 URL: https://issues.apache.org/jira/browse/AIRFLOW-7075 Project: Apache Airflow Issue Type: New Feature Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5664) postgres_to_gcs operator drops milliseconds from timestamps
[
https://issues.apache.org/jira/browse/AIRFLOW-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-5664.
--
Fix Version/s: 2.0.0
Resolution: Done
> postgres_to_gcs operator drops milliseconds from timestamps
> ---
>
> Key: AIRFLOW-5664
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5664
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp, operators
>Affects Versions: 1.10.5
>Reporter: Joseph
>Assignee: Oluwafemi Sule
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Postgres stores timestamps with microsecond resolution. When using the
> postgres_to_gcs operator, timestamps are converted to epoch/unix time using
> the datetime.timetuple() method. This method drops the microseconds and so
> you'll end up with a storage object that looks like this:
> {code:java}
> {"id": 1, "last_modified": 1571038537.0}
> {"id": 2, "last_modified": 1571038537.0}
> {"id": 3, "last_modified": 1571038537.0}
> {code}
> When it should look like this:
> {code:java}
> {"id": 1, "last_modified": 1571038537.123}
> {"id": 2, "last_modified": 1571038537.400}
> {"id": 3, "last_modified": 1571038537.455}
> {code}
> It would be useful to keep the timestamps' full resolution.
> I believe the same issue may occur with airflow.operators.mysql_to_gcs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7051) Implement Vertica Hook Tests
[ https://issues.apache.org/jira/browse/AIRFLOW-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-7051. -- Fix Version/s: 2.0.0 Resolution: Done > Implement Vertica Hook Tests > > > Key: AIRFLOW-7051 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7051 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.9 >Reporter: Sayed Mohammad Hossein Torabi >Assignee: Sayed Mohammad Hossein Torabi >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6975) Base AWSHook AssumeRoleWithSAML
[ https://issues.apache.org/jira/browse/AIRFLOW-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6975. -- Fix Version/s: 2.0.0 Resolution: Done > Base AWSHook AssumeRoleWithSAML > --- > > Key: AIRFLOW-6975 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6975 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Bjorn Olsen >Assignee: Bjorn Olsen >Priority: Minor > Fix For: 2.0.0 > > > Base AWS Hook currently does AssumeRole but we require it to additionally be > able to do AssumeRoleWithSAML. > +Current+ > [https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerole] > The AssumeRole API operation is useful for allowing existing IAM users to > access AWS resources that they don't already have access to. > (This requires an AWS IAM user) > +Proposed addition+ > [https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerolewithsaml] > The AssumeRoleWithSAML API operation returns a set of temporary security > credentials for federated users who are authenticated by your organization's > existing identity system. > (This allows federated login using another IDP rather than requiring an AWS > IAM user). > > +Use case+ > We need to be able to authenticate an AD user against our IDP (Windows Active > Directory). > We can obtain a SAML assertion from our IDP, and then provide it to AWS STS > to exchange it for AWS temporary credentials, thus authorising us to use AWS > services. > The AWS AssumeRoleWithSAML API is intended for this use case, and the Base > AWS Hook should be updated to allow for this method of authentication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6954) Use DagRunType instead of ID_PREFIX in run_id
[ https://issues.apache.org/jira/browse/AIRFLOW-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6954. -- Fix Version/s: 2.0.0 Resolution: Done > Use DagRunType instead of ID_PREFIX in run_id > - > > Key: AIRFLOW-6954 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6954 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6980) Improve system tests and building providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6980. -- Fix Version/s: 2.0.0 Resolution: Done > Improve system tests and building providers package > --- > > Key: AIRFLOW-6980 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6980 > Project: Apache Airflow > Issue Type: Improvement > Components: packages >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6990) Improve system tests for Google Marketing Platform
[ https://issues.apache.org/jira/browse/AIRFLOW-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6990. -- Fix Version/s: 2.0.0 Resolution: Done > Improve system tests for Google Marketing Platform > -- > > Key: AIRFLOW-6990 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6990 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6984) Improve setup/teardown in SFTP-GCS system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6984. -- Fix Version/s: 2.0.0 Resolution: Done > Improve setup/teardown in SFTP-GCS system tests > --- > > Key: AIRFLOW-6984 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6984 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6990) Improve system tests for Google Marketing Platform
Tomasz Urbaszek created AIRFLOW-6990: Summary: Improve system tests for Google Marketing Platform Key: AIRFLOW-6990 URL: https://issues.apache.org/jira/browse/AIRFLOW-6990 Project: Apache Airflow Issue Type: Improvement Components: gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6915) Adds AI Platform Console link for MLEngineStartTrainingJobOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6915. -- Fix Version/s: 2.0.0 Resolution: Done > Adds AI Platform Console link for MLEngineStartTrainingJobOperator > -- > > Key: AIRFLOW-6915 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6915 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Mark Zhang >Assignee: Mark Zhang >Priority: Major > Fix For: 2.0.0 > > > Adds AI Platform Console link for MLEngineStartTrainingJobOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051948#comment-17051948 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Hi [~toddy86], any update? > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6984) Improve setup/teardown in SFTP-GCS system tests
Tomasz Urbaszek created AIRFLOW-6984: Summary: Improve setup/teardown in SFTP-GCS system tests Key: AIRFLOW-6984 URL: https://issues.apache.org/jira/browse/AIRFLOW-6984 Project: Apache Airflow Issue Type: Improvement Components: gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6980) Improve system tests and building providers package
Tomasz Urbaszek created AIRFLOW-6980: Summary: Improve system tests and building providers package Key: AIRFLOW-6980 URL: https://issues.apache.org/jira/browse/AIRFLOW-6980 Project: Apache Airflow Issue Type: Improvement Components: backport-packages, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6971) Fix return type in CloudSpeechToTextRecognizeSpeechOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6971. -- Fix Version/s: 2.0.0 Resolution: Fixed > Fix return type in CloudSpeechToTextRecognizeSpeechOperator > > > Key: AIRFLOW-6971 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6971 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6977) Fix BigQuery DTS example DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6977: - Summary: Fix BigQuery DTS example DAG (was: Fix BigQuery example DAG) > Fix BigQuery DTS example DAG > > > Key: AIRFLOW-6977 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6977 > Project: Apache Airflow > Issue Type: Improvement > Components: examples >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6977) Fix BigQuery example DAG
Tomasz Urbaszek created AIRFLOW-6977: Summary: Fix BigQuery example DAG Key: AIRFLOW-6977 URL: https://issues.apache.org/jira/browse/AIRFLOW-6977 Project: Apache Airflow Issue Type: Improvement Components: examples Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6973) Make GCSCreateBucketOperator idempotent
Tomasz Urbaszek created AIRFLOW-6973: Summary: Make GCSCreateBucketOperator idempotent Key: AIRFLOW-6973 URL: https://issues.apache.org/jira/browse/AIRFLOW-6973 Project: Apache Airflow Issue Type: Improvement Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6971) Fix return type in CloudSpeechToTextRecognizeSpeechOperator
Tomasz Urbaszek created AIRFLOW-6971: Summary: Fix return type in CloudSpeechToTextRecognizeSpeechOperator Key: AIRFLOW-6971 URL: https://issues.apache.org/jira/browse/AIRFLOW-6971 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6970) Improve GCP Video Intelligence system tests
Tomasz Urbaszek created AIRFLOW-6970: Summary: Improve GCP Video Intelligence system tests Key: AIRFLOW-6970 URL: https://issues.apache.org/jira/browse/AIRFLOW-6970 Project: Apache Airflow Issue Type: Improvement Components: gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6958) Fix Intermittent CI failure
[
https://issues.apache.org/jira/browse/AIRFLOW-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-6958.
--
Resolution: Fixed
> Fix Intermittent CI failure
> ---
>
> Key: AIRFLOW-6958
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6958
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> CI is failing because list can have different order
> https://travis-ci.org/kaxil/airflow/jobs/656542706#L1883-L1892
> {code:python}
> ti_to_schedule = []
> dag_file_processor._process_task_instances(dag,
> task_instances_list=ti_to_schedule)
>
> > assert ti_to_schedule == [
> (dag.dag_id, dag_task1.task_id, DEFAULT_DATE, TRY_NUMBER),
> (dag.dag_id, dag_task2.task_id, DEFAULT_DATE, TRY_NUMBER),
> ]
> E AssertionError: assert [('test_sched...0, STD]>), 1)] ==
> [('test_sched...e [UTC]>), 1)]
> E At index 0 diff:
> ('test_scheduler_process_execute_task_depends_on_past', 'dummy2',
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo= +00:00:00, STD]>), 1) !=
> ('test_scheduler_process_execute_task_depends_on_past', 'dummy1',
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo=), 1)
> E Use -v to get the full diff{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6924) Fix Google DLP operators return types
[ https://issues.apache.org/jira/browse/AIRFLOW-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6924. -- Fix Version/s: 2.0.0 Resolution: Fixed > Fix Google DLP operators return types > - > > Key: AIRFLOW-6924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6924 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6954) Use DagRunType instead of ID_PREFIX in run_id
Tomasz Urbaszek created AIRFLOW-6954: Summary: Use DagRunType instead of ID_PREFIX in run_id Key: AIRFLOW-6954 URL: https://issues.apache.org/jira/browse/AIRFLOW-6954 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6948) Remove ASCII Airflow from version command
Tomasz Urbaszek created AIRFLOW-6948: Summary: Remove ASCII Airflow from version command Key: AIRFLOW-6948 URL: https://issues.apache.org/jira/browse/AIRFLOW-6948 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6928) Update Google Dataproc example and system test
Tomasz Urbaszek created AIRFLOW-6928: Summary: Update Google Dataproc example and system test Key: AIRFLOW-6928 URL: https://issues.apache.org/jira/browse/AIRFLOW-6928 Project: Apache Airflow Issue Type: Improvement Components: examples, gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6927) Update Google vision system tests
Tomasz Urbaszek created AIRFLOW-6927: Summary: Update Google vision system tests Key: AIRFLOW-6927 URL: https://issues.apache.org/jira/browse/AIRFLOW-6927 Project: Apache Airflow Issue Type: Improvement Components: gcp, tests Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6926) Fix Google Tasks operators return types and idempotency
Tomasz Urbaszek created AIRFLOW-6926: Summary: Fix Google Tasks operators return types and idempotency Key: AIRFLOW-6926 URL: https://issues.apache.org/jira/browse/AIRFLOW-6926 Project: Apache Airflow Issue Type: Improvement Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6924) Fix Google DLP operators return types
Tomasz Urbaszek created AIRFLOW-6924: Summary: Fix Google DLP operators return types Key: AIRFLOW-6924 URL: https://issues.apache.org/jira/browse/AIRFLOW-6924 Project: Apache Airflow Issue Type: Improvement Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6864) Make airfow/jobs pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6864. -- Fix Version/s: 2.0.0 Resolution: Done > Make airfow/jobs pylint compatible > -- > > Key: AIRFLOW-6864 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6864 > Project: Apache Airflow > Issue Type: Improvement > Components: pylint >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6909) Prepare backport packages on post-test stage
[ https://issues.apache.org/jira/browse/AIRFLOW-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6909. -- Fix Version/s: 1.10.10 Resolution: Done > Prepare backport packages on post-test stage > > > Key: AIRFLOW-6909 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6909 > Project: Apache Airflow > Issue Type: New Feature > Components: ci >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6905) Update pinwheel.svg
[ https://issues.apache.org/jira/browse/AIRFLOW-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6905. -- Fix Version/s: 1.10.10 Resolution: Done > Update pinwheel.svg > --- > > Key: AIRFLOW-6905 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6905 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.10 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6905) Update pinwheel.svg
Tomasz Urbaszek created AIRFLOW-6905: Summary: Update pinwheel.svg Key: AIRFLOW-6905 URL: https://issues.apache.org/jira/browse/AIRFLOW-6905 Project: Apache Airflow Issue Type: Improvement Components: ui Affects Versions: 1.10.10 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6204) Add GCP system tests helper
[ https://issues.apache.org/jira/browse/AIRFLOW-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6204. -- Fix Version/s: 2.0.0 Resolution: Done > Add GCP system tests helper > --- > > Key: AIRFLOW-6204 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6204 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6864) Make airfow/jobs pylint compatible
Tomasz Urbaszek created AIRFLOW-6864: Summary: Make airfow/jobs pylint compatible Key: AIRFLOW-6864 URL: https://issues.apache.org/jira/browse/AIRFLOW-6864 Project: Apache Airflow Issue Type: Improvement Components: pylint Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6863) Make airflow/task and airflow/ti_deps pylint compatible
Tomasz Urbaszek created AIRFLOW-6863: Summary: Make airflow/task and airflow/ti_deps pylint compatible Key: AIRFLOW-6863 URL: https://issues.apache.org/jira/browse/AIRFLOW-6863 Project: Apache Airflow Issue Type: Improvement Components: pylint Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040839#comment-17040839 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- No rush indeed, let me know if you need any help or review :) > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6759) Missing cancel job operator/hook for MLEngine
[ https://issues.apache.org/jira/browse/AIRFLOW-6759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6759. -- Fix Version/s: 2.0.0 Resolution: Implemented > Missing cancel job operator/hook for MLEngine > - > > Key: AIRFLOW-6759 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6759 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, hooks, operators >Affects Versions: 2.0.0 >Reporter: Mark Zhang >Assignee: Mark Zhang >Priority: Minor > Fix For: 2.0.0 > > > Missing cancel training job operator and its associated hook for GCP MLEngine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6405) Bigquery Update Table Properties Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6405. -- Fix Version/s: 2.0.0 Resolution: Done > Bigquery Update Table Properties Operator > - > > Key: AIRFLOW-6405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6405 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 1.10.7 >Reporter: Jithin Sukumar >Assignee: Jithin Sukumar >Priority: Minor > Fix For: 2.0.0 > > > Currently, Airflow doesn't seem to support BigQuery update table operations > [1] (specifically to update the properties of a table). Is this already under > development? > (The use case is conditionally updating the `expiration time` of BQ tables.) > > References: > [1]: [https://cloud.google.com/bigquery/docs/managing-tables] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6801) Make use of timestamp field in ImportError object
[ https://issues.apache.org/jira/browse/AIRFLOW-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6801. -- Fix Version/s: 1.10.10 Resolution: Done > Make use of timestamp field in ImportError object > - > > Key: AIRFLOW-6801 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6801 > Project: Apache Airflow > Issue Type: Improvement > Components: models >Affects Versions: 1.10.7 >Reporter: Chris McLennon >Assignee: Chris McLennon >Priority: Trivial > Fix For: 1.10.10 > > > The ImportError object has a `timestamp` field that goes unused. I believe > this field is intended to report when an ImportError has been inserted into > the table(?). When we add a row to the import_error table it seems that we > should add the current time as the timestamp field as a result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6765) Test full DAGs with airflow dags test
[ https://issues.apache.org/jira/browse/AIRFLOW-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6765. -- Resolution: Done > Test full DAGs with airflow dags test > - > > Key: AIRFLOW-6765 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6765 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > We now have a DebugExecutor for running a full DAG. Similar to airflow test > (airflow tasks test in 2.0), it would be nice to have a command "airflow dags > test", for running a full DAG with the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6765) Test full DAGs with airflow dags test
[ https://issues.apache.org/jira/browse/AIRFLOW-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6765: - Fix Version/s: 2.0.0 > Test full DAGs with airflow dags test > - > > Key: AIRFLOW-6765 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6765 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > We now have a DebugExecutor for running a full DAG. Similar to airflow test > (airflow tasks test in 2.0), it would be nice to have a command "airflow dags > test", for running a full DAG with the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6531. -- Fix Version/s: 2.0.0 Resolution: Done > Add Yandex.Cloud support in Airflow > --- > > Key: AIRFLOW-6531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6531 > Project: Apache Airflow > Issue Type: New Feature > Components: authentication, contrib, examples >Affects Versions: 1.10.7 >Reporter: Peter Volkov >Assignee: Peter Volkov >Priority: Major > Fix For: 2.0.0 > > > Hello there. > I'm here to contribute the support for Yandex.Cloud > ([https://cloud.yandex.com/docs] yet another cloud provider, such as GCP and > AWS) features. > So this issue is created for further PR. > Things will be added: > # New connection type for Yandex.Cloud for authentication. > # Yandex.Cloud hooks to interact with Yandex.Cloud API using GRPC with > [https://github.com/yandex-cloud/python-sdk] module > # Yandex.Cloud operators to create DAGs > # Tests > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6590) Use batch db operations in jobs
[ https://issues.apache.org/jira/browse/AIRFLOW-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6590. -- Fix Version/s: 1.10.10 Resolution: Done > Use batch db operations in jobs > --- > > Key: AIRFLOW-6590 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6590 > Project: Apache Airflow > Issue Type: New Feature > Components: logging >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Tomasz Urbaszek >Priority: Minor > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6770) Unable to run specific test using breeze CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-6770.
--
Fix Version/s: 1.10.10
Resolution: Fixed
> Unable to run specific test using breeze CLI
>
>
> Key: AIRFLOW-6770
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6770
> Project: Apache Airflow
> Issue Type: Bug
> Components: breeze
>Affects Versions: 1.10.9
>Reporter: Jithin Sukumar
>Assignee: Jithin Sukumar
>Priority: Minor
> Fix For: 1.10.10
>
>
> {code:bash}
> ./breeze -t tests/test_core.py
> {code}
> is supposed to run only tests/test_core.py but instead runs all tests within
> tests/ directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6761) Wrong "Workgroup" param in aws_athena_hook.py
[
https://issues.apache.org/jira/browse/AIRFLOW-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-6761.
--
Fix Version/s: 1.10.10
Resolution: Fixed
> Wrong "Workgroup" param in aws_athena_hook.py
> -
>
> Key: AIRFLOW-6761
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6761
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.8, 1.10.9
>Reporter: Van-Duyet Le
>Assignee: Van-Duyet Le
>Priority: Major
> Fix For: 1.10.10
>
>
> Maybe a typo error at run_query function in aws_athena_hook.py
> {code:java}
> Unknown parameter in input: "Workgroup", must be one of: QueryString,
> ClientRequestToken, QueryExecutionContext, ResultConfiguration, WorkGroup
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
> 966, in _run_raw_task
> result = task_copy.execute(context=context)
> File "/opt/airflow/dags/operators/athena_result_operator.py", line 14, in
> execute
> query_execution_id = super().execute(context)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/aws_athena_operator.py",
> line 85, in execute
> self.workgroup)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/hooks/aws_athena_hook.py",
> line 77, in run_query
> Workgroup=workgroup)
> File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276,
> in _api_call
> return self._make_api_call(operation_name, kwargs)
> File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 559,
> in _make_api_call
> api_params, operation_model, context=request_context)
> File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 607,
> in _convert_to_request_dict
> api_params, operation_model)
> File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line
> 297, in serialize_to_request
> raise ParamValidationError(report=report.generate_report())
> botocore.exceptions.ParamValidationError: Parameter validation failed:
> Unknown parameter in input: "Workgroup", must be one of: QueryString,
> ClientRequestToken, QueryExecutionContext, ResultConfiguration, WorkGroup
> {code}
> According to [boto3's
> docs|https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/athena.html#Athena.Client.start_query_execution],
> it's should be {{WorkGroup}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032229#comment-17032229 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Great! I wanted to add GoogleAdsGetAccounts ([https://developers.google.com/google-ads/api/docs/account-management/listing-accounts)] and GoogleAdsGetReportOperator. Do you provide authentication using yaml config file? ** > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032221#comment-17032221 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- [~toddy86] are you going to work on this? I am happy to help :) > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6732) Add Google Ads operators
Tomasz Urbaszek created AIRFLOW-6732: Summary: Add Google Ads operators Key: AIRFLOW-6732 URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 Project: Apache Airflow Issue Type: New Feature Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6676) Add GCSDeleteBucketOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6676. -- Fix Version/s: 2.0.0 Resolution: Done > Add GCSDeleteBucketOperator > --- > > Key: AIRFLOW-6676 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6676 > Project: Apache Airflow > Issue Type: New Feature > Components: api >Affects Versions: 1.10.8 >Reporter: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6590) Add SQL queries logging
[ https://issues.apache.org/jira/browse/AIRFLOW-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek reassigned AIRFLOW-6590: Assignee: Tomasz Urbaszek > Add SQL queries logging > --- > > Key: AIRFLOW-6590 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6590 > Project: Apache Airflow > Issue Type: New Feature > Components: logging >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6707) Simplify Connection.get_hook method
[ https://issues.apache.org/jira/browse/AIRFLOW-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6707: - Fix Version/s: 1.10.8 > Simplify Connection.get_hook method > --- > > Key: AIRFLOW-6707 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6707 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6707) Simplify Connection.get_hook method
[ https://issues.apache.org/jira/browse/AIRFLOW-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6707. -- Resolution: Done > Simplify Connection.get_hook method > --- > > Key: AIRFLOW-6707 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6707 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-3349) StreamLogWriter encoding is wrong type
[ https://issues.apache.org/jira/browse/AIRFLOW-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-3349. -- Fix Version/s: 1.10.8 Resolution: Fixed > StreamLogWriter encoding is wrong type > -- > > Key: AIRFLOW-3349 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3349 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Affects Versions: 1.10.0 >Reporter: Christopher >Priority: Minor > Fix For: 1.10.8 > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > io handlers have encoding that is specified by a string (ie 'UTF-8') or None. > The StreamLogWriter class's encoding attribute is boolean. > > Specifically prevents import of `sympy` from within a PythonOperator, but I > would argue that this implementation should be fixed just so that it can > adhere to the io norm. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6424) Creating a new Airflow operator to modify an existing EMR cluster.
[ https://issues.apache.org/jira/browse/AIRFLOW-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6424. -- Fix Version/s: 2.0.0 Resolution: Done > Creating a new Airflow operator to modify an existing EMR cluster. > -- > > Key: AIRFLOW-6424 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6424 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Affects Versions: 2.0.0 >Reporter: Gavin >Assignee: Gavin >Priority: Minor > Fix For: 2.0.0 > > > Airflow does not have an operator to change the properties of or modify an > existing EMR cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6617) Add tests to PYTHONPATH in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6617. -- Resolution: Abandoned > Add tests to PYTHONPATH in breeze > - > > Key: AIRFLOW-6617 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6617 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6493) RedisHook: Allow SSL connection configuration through "extra" fields
[ https://issues.apache.org/jira/browse/AIRFLOW-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6493: - Fix Version/s: (was: 1.10.8) 2.0.0 > RedisHook: Allow SSL connection configuration through "extra" fields > > > Key: AIRFLOW-6493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6493 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Affects Versions: 1.10.7 >Reporter: Alan Mai >Assignee: Alan Mai >Priority: Minor > Fix For: 2.0.0 > > > RedisHook uses redis-py which has support for SSL connections, but the > current hook has no way of configuring these option. > > Suggestion is to add these options to extra config. > > TLS/SSL support in redis-py is roughly documented in this issue: > [https://github.com/andymccurdy/redis-py/issues/780] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6493) RedisHook: Allow SSL connection configuration through "extra" fields
[ https://issues.apache.org/jira/browse/AIRFLOW-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6493. -- Fix Version/s: 1.10.8 Resolution: Done > RedisHook: Allow SSL connection configuration through "extra" fields > > > Key: AIRFLOW-6493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6493 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Affects Versions: 1.10.7 >Reporter: Alan Mai >Assignee: Alan Mai >Priority: Minor > Fix For: 1.10.8 > > > RedisHook uses redis-py which has support for SSL connections, but the > current hook has no way of configuring these option. > > Suggestion is to add these options to extra config. > > TLS/SSL support in redis-py is roughly documented in this issue: > [https://github.com/andymccurdy/redis-py/issues/780] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6591) Add cli option to stop celery worker
[ https://issues.apache.org/jira/browse/AIRFLOW-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6591. -- Fix Version/s: 2.0.0 Resolution: Done > Add cli option to stop celery worker > > > Key: AIRFLOW-6591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6591 > Project: Apache Airflow > Issue Type: New Feature > Components: celery, cli >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6619) Add cluster_fields to BigQueryCreateEmptyTableOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6619. -- Resolution: Done > Add cluster_fields to BigQueryCreateEmptyTableOperator > -- > > Key: AIRFLOW-6619 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6619 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: master >Reporter: Kamil Gałuszka >Assignee: Kamil Gałuszka >Priority: Major > Fix For: 2.0.0 > > Original Estimate: 1.5h > Remaining Estimate: 1.5h > > cluster_fields is available in BigQueryHook but it's unavailable in > BigQueryCreateEmptyTableOperator. > This ticket is to add that functionality into existing operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6624) Improve webserver command with pidfile checking
Tomasz Urbaszek created AIRFLOW-6624: Summary: Improve webserver command with pidfile checking Key: AIRFLOW-6624 URL: https://issues.apache.org/jira/browse/AIRFLOW-6624 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6620) Mock celery in worker cli test
Tomasz Urbaszek created AIRFLOW-6620: Summary: Mock celery in worker cli test Key: AIRFLOW-6620 URL: https://issues.apache.org/jira/browse/AIRFLOW-6620 Project: Apache Airflow Issue Type: Improvement Components: celery, tests Affects Versions: 1.10.8 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6617) Add tests to PYTHONPATH in breeze
Tomasz Urbaszek created AIRFLOW-6617: Summary: Add tests to PYTHONPATH in breeze Key: AIRFLOW-6617 URL: https://issues.apache.org/jira/browse/AIRFLOW-6617 Project: Apache Airflow Issue Type: Improvement Components: breeze, tests Affects Versions: 1.10.8 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6612) Use builtin method to run flower instead of executing cmd
Tomasz Urbaszek created AIRFLOW-6612: Summary: Use builtin method to run flower instead of executing cmd Key: AIRFLOW-6612 URL: https://issues.apache.org/jira/browse/AIRFLOW-6612 Project: Apache Airflow Issue Type: Improvement Components: celery, cli Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-6145) Rename local_to_gcs service
[ https://issues.apache.org/jira/browse/AIRFLOW-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek reopened AIRFLOW-6145: -- > Rename local_to_gcs service > --- > > Key: AIRFLOW-6145 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6145 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > > Added these classes: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6145) Rename local_to_gcs service
[ https://issues.apache.org/jira/browse/AIRFLOW-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6145. -- Fix Version/s: 2.0.0 Resolution: Done > Rename local_to_gcs service > --- > > Key: AIRFLOW-6145 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6145 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > > Added these classes: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6591) Add cli option to stop celery worker
Tomasz Urbaszek created AIRFLOW-6591: Summary: Add cli option to stop celery worker Key: AIRFLOW-6591 URL: https://issues.apache.org/jira/browse/AIRFLOW-6591 Project: Apache Airflow Issue Type: New Feature Components: celery, cli Affects Versions: 1.10.8 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6590) Add SQL queries logging
Tomasz Urbaszek created AIRFLOW-6590: Summary: Add SQL queries logging Key: AIRFLOW-6590 URL: https://issues.apache.org/jira/browse/AIRFLOW-6590 Project: Apache Airflow Issue Type: New Feature Components: logging Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6541) Use EmrJobFlowSensor for other states
[ https://issues.apache.org/jira/browse/AIRFLOW-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6541: - Fix Version/s: (was: 1.10.8) 2.0.0 > Use EmrJobFlowSensor for other states > - > > Key: AIRFLOW-6541 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6541 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, contrib >Affects Versions: 1.10.7 >Reporter: Mustafa Gök >Assignee: Mustafa Gök >Priority: Minor > Fix For: 2.0.0 > > > Currently, EmrJobFlowSensor waits cluster to be terminated (state: > "TERMINATED"). It can be used for other purposes if cluster states to be > checked (i.e., NON_TERMINAL_STATES) can be passed to the operator. > For example, this sensor could be used to wait cluster to be ready (state: > "RUNNING" or "WAITING"). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6558) Campaign Manager Operators to insert and modify conversions
[ https://issues.apache.org/jira/browse/AIRFLOW-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek updated AIRFLOW-6558: - Summary: Campaign Manager Operators to insert and modify conversions (was: Campaign Manager Operators to insert and modify conversations) > Campaign Manager Operators to insert and modify conversions > --- > > Key: AIRFLOW-6558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6558 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6329) Add AirflowExecutor and native worker
[ https://issues.apache.org/jira/browse/AIRFLOW-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6329. -- Resolution: Abandoned > Add AirflowExecutor and native worker > - > > Key: AIRFLOW-6329 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6329 > Project: Apache Airflow > Issue Type: Bug > Components: executors >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6571) Rewrite BigQueryExecuteQueryOperator to use python client
Tomasz Urbaszek created AIRFLOW-6571: Summary: Rewrite BigQueryExecuteQueryOperator to use python client Key: AIRFLOW-6571 URL: https://issues.apache.org/jira/browse/AIRFLOW-6571 Project: Apache Airflow Issue Type: Improvement Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6146) Rename gcs_to_bq service
[ https://issues.apache.org/jira/browse/AIRFLOW-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6146. -- Fix Version/s: 2.0.0 Resolution: Done > Rename gcs_to_bq service > > > Key: AIRFLOW-6146 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6146 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > > Added these classes: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6558) Campaign Manager Operators to insert and modify conversations
Tomasz Urbaszek created AIRFLOW-6558: Summary: Campaign Manager Operators to insert and modify conversations Key: AIRFLOW-6558 URL: https://issues.apache.org/jira/browse/AIRFLOW-6558 Project: Apache Airflow Issue Type: New Feature Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6534) BigQuery - move methods from BigQueryBaseCursor to BigQueryHook
[ https://issues.apache.org/jira/browse/AIRFLOW-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6534. -- Fix Version/s: 2.0.0 Resolution: Done > BigQuery - move methods from BigQueryBaseCursor to BigQueryHook > --- > > Key: AIRFLOW-6534 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6534 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, operators >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6107) Rename container service
[ https://issues.apache.org/jira/browse/AIRFLOW-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6107. -- Fix Version/s: 2.0.0 Resolution: Done > Rename container service > > > Key: AIRFLOW-6107 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6107 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > > Added these classes: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6548) Restore GCS tests removed by migration
[ https://issues.apache.org/jira/browse/AIRFLOW-6548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6548. -- Fix Version/s: 2.0.0 Resolution: Fixed > Restore GCS tests removed by migration > -- > > Key: AIRFLOW-6548 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6548 > Project: Apache Airflow > Issue Type: Bug > Components: gcp, tests >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6102) Rename dataproc service
[ https://issues.apache.org/jira/browse/AIRFLOW-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6102. -- Fix Version/s: 2.0.0 Resolution: Done > Rename dataproc service > --- > > Key: AIRFLOW-6102 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6102 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Michał Słowikowski >Assignee: Michał Słowikowski >Priority: Minor > Fix For: 2.0.0 > > > Added these classes: > * airflow.gcp.operators.dataproc.DataprocCreateClusterOperator > * airflow.gcp.operators.dataproc.DataprocDeleteClusterOperator > * airflow.gcp.operators.dataproc.DataprocScaleClusterOperator > * airflow.gcp.operators.dataproc.DataprocSubmitHadoopJobOperator > * airflow.gcp.operators.dataproc.DataprocSubmitHiveJobOperator > * airflow.gcp.operators.dataproc.DataprocJobBaseOperator > * airflow.gcp.operators.dataproc.DataprocOperationBaseOperator > * airflow.gcp.operators.dataproc.DataprocSubmitPigJobOperator > * airflow.gcp.operators.dataproc.DataprocSubmitPySparkJobOperator > * airflow.gcp.operators.dataproc.DataprocSubmitSparkJobOperator > * airflow.gcp.operators.dataproc.DataprocSubmitSparkSqlJobOperator > * > airflow.gcp.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator > * airflow.gcp.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
