[jira] [Commented] (AIRFLOW-2571) Improve airflow backfill CLI status display
[ https://issues.apache.org/jira/browse/AIRFLOW-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504283#comment-16504283 ] Chao-Han Tsai commented on AIRFLOW-2571: we can use https://python-enlighten.readthedocs.io/en/latest/ > Improve airflow backfill CLI status display > --- > > Key: AIRFLOW-2571 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2571 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Improve airflow backfill to show percentage completed like: > 100% || -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2574) initdb fails when mysql password contains percent sign
Zihao Zhang created AIRFLOW-2574: Summary: initdb fails when mysql password contains percent sign Key: AIRFLOW-2574 URL: https://issues.apache.org/jira/browse/AIRFLOW-2574 Project: Apache Airflow Issue Type: Bug Components: db Reporter: Zihao Zhang [db.py|https://github.com/apache/incubator-airflow/blob/3358551c8e73d9019900f7a85f18ebfd88591450/airflow/utils/db.py#L345] uses [config.set_main_option|http://alembic.zzzcomputing.com/en/latest/api/config.html#alembic.config.Config.set_main_option] which says "A raw percent sign not part of an interpolation symbol must therefore be escaped" When there is a percent sign in database connection string, this will crash due to bad interpolation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2561] Fix typo in EmailOperator
Repository: incubator-airflow
Updated Branches:
refs/heads/master b8487cd44 -> 3358551c8
[AIRFLOW-2561] Fix typo in EmailOperator
There's a typo in the params in send_email.
It should be mime_charset instead of mine_charset.
Closes #3468 from wolfier/AIRFLOW-2561
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3358551c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3358551c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3358551c
Branch: refs/heads/master
Commit: 3358551c8e73d9019900f7a85f18ebfd88591450
Parents: b8487cd
Author: Alan Ma
Authored: Wed Jun 6 16:25:21 2018 -0700
Committer: r39132
Committed: Wed Jun 6 16:25:21 2018 -0700
--
airflow/operators/email_operator.py| 2 +-
tests/operators/test_email_operator.py | 65 +
2 files changed, 66 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3358551c/airflow/operators/email_operator.py
--
diff --git a/airflow/operators/email_operator.py
b/airflow/operators/email_operator.py
index 4f5396f..ae176ca 100644
--- a/airflow/operators/email_operator.py
+++ b/airflow/operators/email_operator.py
@@ -75,4 +75,4 @@ class EmailOperator(BaseOperator):
def execute(self, context):
send_email(self.to, self.subject, self.html_content,
files=self.files, cc=self.cc, bcc=self.bcc,
- mime_subtype=self.mime_subtype,
mine_charset=self.mime_charset)
+ mime_subtype=self.mime_subtype,
mime_charset=self.mime_charset)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3358551c/tests/operators/test_email_operator.py
--
diff --git a/tests/operators/test_email_operator.py
b/tests/operators/test_email_operator.py
new file mode 100644
index 000..a19f450
--- /dev/null
+++ b/tests/operators/test_email_operator.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function, unicode_literals
+
+import datetime
+import mock
+import unittest
+
+from airflow import configuration, DAG
+from airflow.operators.email_operator import EmailOperator
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2016, 1, 1)
+END_DATE = timezone.datetime(2016, 1, 2)
+INTERVAL = datetime.timedelta(hours=12)
+FROZEN_NOW = timezone.datetime(2016, 1, 2, 12, 1, 1)
+
+send_email_test = mock.Mock()
+
+
+class TestEmailOperator(unittest.TestCase):
+
+def setUp(self):
+super(TestEmailOperator, self).setUp()
+configuration.load_test_config()
+self.dag = DAG(
+'test_dag',
+default_args={
+'owner': 'airflow',
+'start_date': DEFAULT_DATE},
+schedule_interval=INTERVAL)
+self.addCleanup(self.dag.clear)
+
+def _run_as_operator(self, **kwargs):
+task = EmailOperator(
+to='airf...@example.com',
+subject='Test Run',
+html_content='The quick brown fox jumps over the lazy dog',
+task_id='task',
+dag=self.dag,
+**kwargs)
+task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+
+def test_execute(self):
+configuration.conf.set('email', 'EMAIL_BACKEND',
+
'tests.operators.test_email_operator.send_email_test')
+self._run_as_operator()
+send_email_test.assert_called_once()
[jira] [Commented] (AIRFLOW-2561) Typo in Email operator call to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504043#comment-16504043 ] ASF subversion and git services commented on AIRFLOW-2561: -- Commit 3358551c8e73d9019900f7a85f18ebfd88591450 in incubator-airflow's branch refs/heads/master from Alan Ma [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3358551 ] [AIRFLOW-2561] Fix typo in EmailOperator There's a typo in the params in send_email. It should be mime_charset instead of mine_charset. Closes #3468 from wolfier/AIRFLOW-2561 > Typo in Email operator call to send_email > - > > Key: AIRFLOW-2561 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2561 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alan Ma >Assignee: Alan Ma >Priority: Critical > Fix For: 2.0.0 > > > There's a typo in the params in send_email. It should be mime_charset instead > of mine_charset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2561) Typo in Email operator call to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504044#comment-16504044 ] ASF subversion and git services commented on AIRFLOW-2561: -- Commit 3358551c8e73d9019900f7a85f18ebfd88591450 in incubator-airflow's branch refs/heads/master from Alan Ma [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3358551 ] [AIRFLOW-2561] Fix typo in EmailOperator There's a typo in the params in send_email. It should be mime_charset instead of mine_charset. Closes #3468 from wolfier/AIRFLOW-2561 > Typo in Email operator call to send_email > - > > Key: AIRFLOW-2561 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2561 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alan Ma >Assignee: Alan Ma >Priority: Critical > Fix For: 2.0.0 > > > There's a typo in the params in send_email. It should be mime_charset instead > of mine_charset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2561) Typo in Email operator call to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-2561. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3468 [https://github.com/apache/incubator-airflow/pull/3468] > Typo in Email operator call to send_email > - > > Key: AIRFLOW-2561 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2561 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alan Ma >Assignee: Alan Ma >Priority: Critical > Fix For: 2.0.0 > > > There's a typo in the params in send_email. It should be mime_charset instead > of mine_charset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-2573) Cast TIMESTAMP field to float rather than int
[
https://issues.apache.org/jira/browse/AIRFLOW-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-2573 started by Hongyi Wang.
> Cast TIMESTAMP field to float rather than int
> -
>
> Key: AIRFLOW-2573
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2573
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Hongyi Wang
>Assignee: Hongyi Wang
>Priority: Blocker
>
> In current bigquery_hook.py, we have a `_bq_cast(string_field, bq_type)`
> function that help casts a BigQuery row to the appropriate data types.
> {quote}elif bq_type == 'INTEGER' or bq_type == 'TIMESTAMP':
> return int(string_field)
> {quote}
> However, when a bq_type equals to 'TIMESTAMP', it causes ValueError.
> {quote}>>> int('1.458668898E9')
> ValueError: invalid literal for int() with base 10: '1.458668898E9'
> {quote}
> Because 'TIMESTAMP' in bigquery is stored as double in python, thus should be
> cast to float instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2573) Cast TIMESTAMP field to float rather than int
Hongyi Wang created AIRFLOW-2573:
Summary: Cast TIMESTAMP field to float rather than int
Key: AIRFLOW-2573
URL: https://issues.apache.org/jira/browse/AIRFLOW-2573
Project: Apache Airflow
Issue Type: Bug
Reporter: Hongyi Wang
Assignee: Hongyi Wang
In current bigquery_hook.py, we have a `_bq_cast(string_field, bq_type)`
function that help casts a BigQuery row to the appropriate data types.
{quote}elif bq_type == 'INTEGER' or bq_type == 'TIMESTAMP':
return int(string_field)
{quote}
However, when a bq_type equals to 'TIMESTAMP', it causes ValueError.
{quote}>>> int('1.458668898E9')
ValueError: invalid literal for int() with base 10: '1.458668898E9'
{quote}
Because 'TIMESTAMP' in bigquery is stored as double in python, thus should be
cast to float instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2572) Airflow /admin/configurationview/ Running Configuration not accurate
Joe Jasinski created AIRFLOW-2572:
-
Summary: Airflow /admin/configurationview/ Running Configuration
not accurate
Key: AIRFLOW-2572
URL: https://issues.apache.org/jira/browse/AIRFLOW-2572
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.9.1
Environment: Debian 8 (in a Docker container)
Reporter: Joe Jasinski
The admin Runtime Configuration section of /admin/configurationview/ is showing
config settings from the config file but not the environment variables set.
When that page renders, this loop is called, which uses the config.as_dict()
method:
https://github.com/apache/incubator-airflow/blob/master/airflow/www/views.py#L2868
The conf.as_dict() method doesn't properly read in from the environment
variables. In the example below, see how we can get the expected value from
conf.get() but not from conf.as_dict(). I think it has to do with the
implementation of as_dict()
export
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:@postgres:5432/airflow
>>> from airflow import configuration as conf
>>> import os
>>> os.environ["AIRFLOW__CORE__SQL_ALCHEMY_CONN"]
'postgresql+psycopg2://airflow:@postgres:5432/airflow'
>>> dict(conf.as_dict(True, True))['core']['sql_alchemy_conn']
('sqlite:usr/local/airflow/airflow/airflow.db', 'bash cmd')
>>> conf.get('core', 'sql_alchemy_conn')
'postgresql+psycopg2://airflow:@postgres:5432/airflow'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2560] Adding support for internalIpOnly to DataprocClusterCreateOperator
Repository: incubator-airflow
Updated Branches:
refs/heads/master 0bc6042ce -> b8487cd44
[AIRFLOW-2560] Adding support for internalIpOnly to
DataprocClusterCreateOperator
Closes #3458 from piffall/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b8487cd4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b8487cd4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b8487cd4
Branch: refs/heads/master
Commit: b8487cd4468fba41f6468c936487ffc2203b4c2e
Parents: 0bc6042
Author: Cristòfol Torrens
Authored: Wed Jun 6 13:06:03 2018 +0100
Committer: Kaxil Naik
Committed: Wed Jun 6 13:06:03 2018 +0100
--
airflow/contrib/operators/dataproc_operator.py | 13 -
.../contrib/operators/test_dataproc_operator.py | 51
2 files changed, 54 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b8487cd4/airflow/contrib/operators/dataproc_operator.py
--
diff --git a/airflow/contrib/operators/dataproc_operator.py
b/airflow/contrib/operators/dataproc_operator.py
index 4973eb1..5d59f7f 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -93,6 +93,10 @@ class DataprocClusterCreateOperator(BaseOperator):
:param subnetwork_uri: The subnetwork uri to be used for machine
communication,
cannot be specified with network_uri
:type subnetwork_uri: string
+:param internal_ip_only: If true, all instances in the cluster will only
+have internal IP addresses. This can only be enabled for subnetwork
+enabled networks
+:type internal_ip_only: bool
:param tags: The GCE tags to add to all instances
:type tags: list[string]
:param region: leave as 'global', might become relevant in the future.
(templated)
@@ -111,7 +115,7 @@ class DataprocClusterCreateOperator(BaseOperator):
A duration in seconds.
:type idle_delete_ttl: int
:param auto_delete_time: The time when cluster will be auto-deleted.
-:type auto_delete_time: datetime
+:type auto_delete_time: datetime.datetime
:param auto_delete_ttl: The life duration of cluster, the cluster will be
auto-deleted at the end of this duration.
A duration in seconds. (If auto_delete_time is set this parameter will
be ignored)
@@ -128,6 +132,7 @@ class DataprocClusterCreateOperator(BaseOperator):
zone,
network_uri=None,
subnetwork_uri=None,
+ internal_ip_only=None,
tags=None,
storage_bucket=None,
init_actions_uris=None,
@@ -173,6 +178,7 @@ class DataprocClusterCreateOperator(BaseOperator):
self.zone = zone
self.network_uri = network_uri
self.subnetwork_uri = subnetwork_uri
+self.internal_ip_only = internal_ip_only
self.tags = tags
self.region = region
self.service_account = service_account
@@ -306,6 +312,11 @@ class DataprocClusterCreateOperator(BaseOperator):
if self.subnetwork_uri:
cluster_data['config']['gceClusterConfig']['subnetworkUri'] = \
self.subnetwork_uri
+if self.internal_ip_only:
+if not self.subnetwork_uri:
+raise AirflowException("Set internal_ip_only to true only when"
+ " you pass a subnetwork_uri.")
+cluster_data['config']['gceClusterConfig']['internalIpOnly'] = True
if self.tags:
cluster_data['config']['gceClusterConfig']['tags'] = self.tags
if self.image_version:
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b8487cd4/tests/contrib/operators/test_dataproc_operator.py
--
diff --git a/tests/contrib/operators/test_dataproc_operator.py
b/tests/contrib/operators/test_dataproc_operator.py
index 6cb2044..65ff5cd 100644
--- a/tests/contrib/operators/test_dataproc_operator.py
+++ b/tests/contrib/operators/test_dataproc_operator.py
@@ -22,7 +22,7 @@ import datetime
import re
import unittest
-from airflow import DAG
+from airflow import DAG, AirflowException
from airflow.contrib.operators.dataproc_operator import \
DataprocClusterCreateOperator, \
DataprocClusterDeleteOperator, \
@@ -55,6 +55,7 @@ NUM_WORKERS = 123
ZONE = 'us-central1-a'
NETWORK_URI = '/projects/project_id/regions/global/net'
SUBNETWORK_URI = '/projects/project_id/regions/global/subnet'
+INTERNAL_IP_ONLY = True
TAGS = ['tag1', 'tag2']
STORAGE_BUCKET = 'gs://airflow-test-bucket/'
IMAGE_VERSION = '1.1'
@@ -98,6
[jira] [Resolved] (AIRFLOW-2560) Add support for GceClusterConfig.internalIpOnly in dataproc create cluster operation.
[ https://issues.apache.org/jira/browse/AIRFLOW-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2560. - Resolution: Fixed Fix Version/s: (was: Airflow 2.0) 2.0.0 Issue resolved by pull request #3458 [https://github.com/apache/incubator-airflow/pull/3458] > Add support for GceClusterConfig.internalIpOnly in dataproc create cluster > operation. > - > > Key: AIRFLOW-2560 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2560 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, operators >Reporter: Cristòfol Torrens >Assignee: Cristòfol Torrens >Priority: Minor > Labels: dataproc, google, google-compute-engine, operator > Fix For: 2.0.0 > > > This is the only missing parameter of GceClusterConfig for Cluster creation. > So lets add it to the operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2571) Improve airflow backfill CLI status display
Chao-Han Tsai created AIRFLOW-2571: -- Summary: Improve airflow backfill CLI status display Key: AIRFLOW-2571 URL: https://issues.apache.org/jira/browse/AIRFLOW-2571 Project: Apache Airflow Issue Type: Improvement Reporter: Chao-Han Tsai Assignee: Chao-Han Tsai Improve airflow backfill to show percentage completed like: 100% || -- This message was sent by Atlassian JIRA (v7.6.3#76005)
