[jira] [Assigned] (AIRFLOW-2476) tabulate update: 0.8.2 is tested

[Ruslan Dautkhanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruslan Dautkhanov reassigned AIRFLOW-2476:
--

Assignee: Ruslan Dautkhanov

> tabulate update: 0.8.2 is tested
> 
>
> Key: AIRFLOW-2476
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2476
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: Airflow 2.0, Airflow 1.8, 1.9.0, 1.10.0, 1.10
>Reporter: Ruslan Dautkhanov
>Assignee: Ruslan Dautkhanov
>Priority: Major
>
> As discussed on the dev list, tabulate==0.8.2 is good to go with Airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2476) tabulate update: 0.8.2 is tested

[Ruslan Dautkhanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruslan Dautkhanov updated AIRFLOW-2476:
---
Affects Version/s: 1.10
   1.10.0
   1.9.0
   Airflow 1.8
   Airflow 2.0

> tabulate update: 0.8.2 is tested
> 
>
> Key: AIRFLOW-2476
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2476
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: Airflow 2.0, Airflow 1.8, 1.9.0, 1.10.0, 1.10
>Reporter: Ruslan Dautkhanov
>Priority: Major
>
> As discussed on the dev list, tabulate==0.8.2 is good to go with Airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2476) tabulate update: 0.8.2 is tested

[Ruslan Dautkhanov (JIRA)]

Ruslan Dautkhanov created AIRFLOW-2476:
--

 Summary: tabulate update: 0.8.2 is tested
 Key: AIRFLOW-2476
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2476
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Ruslan Dautkhanov


As discussed on the dev list, tabulate==0.8.2 is good to go with Airflow.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2475) Add a "go to triggered DAG" button when using TriggerDagRunOperator

[Bas Harenslak (JIRA)]

Bas Harenslak created AIRFLOW-2475:
--

 Summary: Add a "go to triggered DAG" button when using 
TriggerDagRunOperator
 Key: AIRFLOW-2475
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2475
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ui
Reporter: Bas Harenslak
Assignee: Bas Harenslak
 Attachments: image-2018-05-16-22-37-15-891.png

It would be useful to have a button for going to the triggered DAG view when 
using the TriggerDagRunOperator, instead of reading code to know which DAG was 
triggered.

I've implemented this and the modal now looks like this when opening a 
TriggerDagRunOperator. PR coming.

!image-2018-05-16-22-37-15-891.png|width=345,height=253!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2474) Should not attempt to import snakebite in py3

[Joy Gao (JIRA)]

Joy Gao created AIRFLOW-2474:


 Summary: Should not attempt to import snakebite in py3
 Key: AIRFLOW-2474
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2474
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Joy Gao
Assignee: Joy Gao


Patch in HDFSHook module to stop importing snakebite in PY3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2472) Implement MySqlHook.bulk_dump

[Kengo Seki (JIRA)]

Kengo Seki created AIRFLOW-2472:
---

 Summary: Implement MySqlHook.bulk_dump
 Key: AIRFLOW-2472
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2472
 Project: Apache Airflow
  Issue Type: Improvement
  Components: hooks
Reporter: Kengo Seki
Assignee: Kengo Seki


For the purpose of backup, migration, ETL etc., implement 
{{MySqlHook.bulk_dump}} which is opposite to already implemented {{bulk_load}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2471) Fix HiveCliHook.load_df to use unused parameters

[Kengo Seki (JIRA)]

Kengo Seki created AIRFLOW-2471:
---

 Summary: Fix HiveCliHook.load_df to use unused parameters
 Key: AIRFLOW-2471
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2471
 Project: Apache Airflow
  Issue Type: Bug
  Components: hive_hooks, hooks
Reporter: Kengo Seki
Assignee: Kengo Seki


HiveCliHook.load_df has parameters called create and recreate:


{code}
def load_df(
self,
df,
table,
create=True,
recreate=False,

(snip)

:param create: whether to create the table if it doesn't exist
:type create: bool
:param recreate: whether to drop and recreate the table at every
execution
:type recreate: bool
{code}

but these are already used as the default value.　For example, even if 
specifying {{recreate=True}}, {{DROP TABLE}} is not executed before {{CREATE 
TABLE}}.

{code}
In [1]: import pandas as pd

In [2]: from airflow.hooks.hive_hooks import HiveCliHook

In [3]: df = pd.DataFrame({"c": range(0, 10)})

In [4]: h = HiveCliHook()
[2018-05-16 10:27:55,814] {base_hook.py:83} INFO - Using connection to: 
localhost

In [5]: h.load_df(df, "t", recreate=True)
[2018-05-16 10:28:17,351] {hive_hooks.py:424} INFO - CREATE TABLE IF NOT EXISTS 
t (
c BIGINT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS textfile
;
[2018-05-16 10:28:17,353] {hive_hooks.py:217} INFO - beeline -u 
jdbc:hive2://localhost:1/default;auth=none -f 
/tmp/airflow_hiveop__zc0kY/tmp99TFZK
[2018-05-16 10:28:19,730] {hive_hooks.py:232} INFO - Connecting to 
jdbc:hive2://localhost:1/default;auth=none
[2018-05-16 10:28:20,127] {hive_hooks.py:232} INFO - Connected to: Apache Hive 
(version 1.2.1)
[2018-05-16 10:28:20,128] {hive_hooks.py:232} INFO - Driver: Hive JDBC (version 
1.2.1)
[2018-05-16 10:28:20,129] {hive_hooks.py:232} INFO - Transaction isolation: 
TRANSACTION_REPEATABLE_READ
[2018-05-16 10:28:20,205] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> USE default;
[2018-05-16 10:28:20,446] {hive_hooks.py:232} INFO - No rows affected (0.234 
seconds)
[2018-05-16 10:28:20,481] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> CREATE TABLE IF NOT EXISTS t (
[2018-05-16 10:28:20,485] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> c BIGINT)
[2018-05-16 10:28:20,491] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> ROW FORMAT DELIMITED
[2018-05-16 10:28:20,497] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> FIELDS TERMINATED BY ','
[2018-05-16 10:28:20,508] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> STORED AS textfile
[2018-05-16 10:28:20,582] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default> ;No rows affected (0.074 seconds)
[2018-05-16 10:28:20,597] {hive_hooks.py:232} INFO - 0: 
jdbc:hive2://localhost:1/default>
[2018-05-16 10:28:20,598] {hive_hooks.py:232} INFO - Closing: 0: 
jdbc:hive2://localhost:1/default;auth=none

(snip)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-48] Parse connection uri querystring

[fokko]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 802cbd9b9 -> b17a99d71


[AIRFLOW-48] Parse connection uri querystring

Closes #3292 from inytar/extras-in-uri


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b17a99d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b17a99d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b17a99d7

Branch: refs/heads/master
Commit: b17a99d714310cd05577a6d1b3b4baa101aec674
Parents: 802cbd9
Author: inytar 
Authored: Wed May 16 10:53:41 2018 +0200
Committer: Fokko Driesprong 
Committed: Wed May 16 10:53:41 2018 +0200

--
 airflow/models.py |  4 +++-
 tests/models.py   | 24 
 2 files changed, 27 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b17a99d7/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 2322860..c9fee0c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -52,7 +52,7 @@ import hashlib
 
 import uuid
 from datetime import datetime
-from urllib.parse import urlparse, quote
+from urllib.parse import urlparse, quote, parse_qsl
 
 from sqlalchemy import (
 Column, Integer, String, DateTime, Text, Boolean, ForeignKey, PickleType,
@@ -636,6 +636,8 @@ class Connection(Base, LoggingMixin):
 self.login = temp_uri.username
 self.password = temp_uri.password
 self.port = temp_uri.port
+if temp_uri.query:
+self.extra = json.dumps(dict(parse_qsl(temp_uri.query)))
 
 def get_password(self):
 if self._password and self.is_encrypted:

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b17a99d7/tests/models.py
--
diff --git a/tests/models.py b/tests/models.py
index c2e54e5..2bf61fe 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -2326,3 +2326,27 @@ class ConnectionTest(unittest.TestCase):
 mock_get.return_value = 'dGVzdA=='
 test_connection = Connection(extra='testextra')
 self.assertEqual(test_connection.extra, 'testextra')
+
+def test_connection_from_uri_without_extras(self):
+uri = 'scheme://user:password@host%2flocation:1234/schema'
+connection = Connection(uri=uri)
+self.assertEqual(connection.conn_type, 'scheme')
+self.assertEqual(connection.host, 'host/location')
+self.assertEqual(connection.schema, 'schema')
+self.assertEqual(connection.login, 'user')
+self.assertEqual(connection.password, 'password')
+self.assertEqual(connection.port, 1234)
+self.assertIsNone(connection.extra)
+
+def test_connection_from_uri_with_extras(self):
+uri = 'scheme://user:password@host%2flocation:1234/schema?'\
+'extra1=a%20value=%2fpath%2f'
+connection = Connection(uri=uri)
+self.assertEqual(connection.conn_type, 'scheme')
+self.assertEqual(connection.host, 'host/location')
+self.assertEqual(connection.schema, 'schema')
+self.assertEqual(connection.login, 'user')
+self.assertEqual(connection.password, 'password')
+self.assertEqual(connection.port, 1234)
+self.assertDictEqual(connection.extra_dejson, {'extra1': 'a value',
+   'extra2': '/path/'})

[jira] [Resolved] (AIRFLOW-2467) Improve AirflowImporter warn message

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-2467.
---
   Resolution: Fixed
Fix Version/s: 2.0.0
   1.10.0

Issue resolved by pull request #3361
[https://github.com/apache/incubator-airflow/pull/3361]

> Improve AirflowImporter warn message
> 
>
> Key: AIRFLOW-2467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2467
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Dan Fowler
>Assignee: Dan Fowler
>Priority: Trivial
> Fix For: 1.10.0, 2.0.0
>
>
> Currently when importing modules from airflow.operators there is a warning 
> message that looks like this:
> {code:java}
> [2018-05-14 02:43:19,205] {{logging_mixin.py:84}} WARNING - 
> /usr/local/airflow/code/airflow/utils/helpers.py:351: DeprecationWarning: 
> Importing DummyOperator directly from  '/usr/local/airflow/code/airflow/operators/__init__.py'> has been deprecated. 
> Please import from ' '/usr/local/airflow/code/airflow/operators/__init__.py'>.[operator_module]' 
> instead. Support for direct imports will be dropped entirely in Airflow 2.0.
> {code}
> It's not that clear what the issue is, this message should be improved for 
> clarity. Most likely just use the module name rather than the module class's 
> str repr.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2467) Improve AirflowImporter warn message

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477077#comment-16477077
 ] 

ASF subversion and git services commented on AIRFLOW-2467:
--

Commit 802cbd9b9b497a3630aaaf2e6f656a98ffc7d550 in incubator-airflow's branch 
refs/heads/master from [~dfowler]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=802cbd9 ]

[AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module 
name

Closes #3361 from dan-sf/AIRFLOW-2467


> Improve AirflowImporter warn message
> 
>
> Key: AIRFLOW-2467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2467
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Dan Fowler
>Assignee: Dan Fowler
>Priority: Trivial
> Fix For: 1.10.0, 2.0.0
>
>
> Currently when importing modules from airflow.operators there is a warning 
> message that looks like this:
> {code:java}
> [2018-05-14 02:43:19,205] {{logging_mixin.py:84}} WARNING - 
> /usr/local/airflow/code/airflow/utils/helpers.py:351: DeprecationWarning: 
> Importing DummyOperator directly from  '/usr/local/airflow/code/airflow/operators/__init__.py'> has been deprecated. 
> Please import from ' '/usr/local/airflow/code/airflow/operators/__init__.py'>.[operator_module]' 
> instead. Support for direct imports will be dropped entirely in Airflow 2.0.
> {code}
> It's not that clear what the issue is, this message should be improved for 
> clarity. Most likely just use the module name rather than the module class's 
> str repr.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2467) Improve AirflowImporter warn message

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477076#comment-16477076
 ] 

ASF subversion and git services commented on AIRFLOW-2467:
--

Commit 802cbd9b9b497a3630aaaf2e6f656a98ffc7d550 in incubator-airflow's branch 
refs/heads/master from [~dfowler]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=802cbd9 ]

[AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module 
name

Closes #3361 from dan-sf/AIRFLOW-2467


> Improve AirflowImporter warn message
> 
>
> Key: AIRFLOW-2467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2467
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Dan Fowler
>Assignee: Dan Fowler
>Priority: Trivial
> Fix For: 1.10.0, 2.0.0
>
>
> Currently when importing modules from airflow.operators there is a warning 
> message that looks like this:
> {code:java}
> [2018-05-14 02:43:19,205] {{logging_mixin.py:84}} WARNING - 
> /usr/local/airflow/code/airflow/utils/helpers.py:351: DeprecationWarning: 
> Importing DummyOperator directly from  '/usr/local/airflow/code/airflow/operators/__init__.py'> has been deprecated. 
> Please import from ' '/usr/local/airflow/code/airflow/operators/__init__.py'>.[operator_module]' 
> instead. Support for direct imports will be dropped entirely in Airflow 2.0.
> {code}
> It's not that clear what the issue is, this message should be improved for 
> clarity. Most likely just use the module name rather than the module class's 
> str repr.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module name

[fokko]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master ee81a6ad8 -> 802cbd9b9


[AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module 
name

Closes #3361 from dan-sf/AIRFLOW-2467


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/802cbd9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/802cbd9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/802cbd9b

Branch: refs/heads/master
Commit: 802cbd9b9b497a3630aaaf2e6f656a98ffc7d550
Parents: ee81a6a
Author: dan-sf 
Authored: Wed May 16 10:46:27 2018 +0200
Committer: Fokko Driesprong 
Committed: Wed May 16 10:46:27 2018 +0200

--
 airflow/utils/helpers.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/802cbd9b/airflow/utils/helpers.py
--
diff --git a/airflow/utils/helpers.py b/airflow/utils/helpers.py
index 3389788..911890d 100644
--- a/airflow/utils/helpers.py
+++ b/airflow/utils/helpers.py
@@ -338,11 +338,11 @@ class AirflowImporter(object):
 # This functionality is deprecated, and AirflowImporter should be
 # removed in 2.0.
 warnings.warn(
-"Importing {i} directly from {m} has been "
+"Importing '{i}' directly from '{m}' has been "
 "deprecated. Please import from "
 "'{m}.[operator_module]' instead. Support for direct "
 "imports will be dropped entirely in Airflow 2.0.".format(
-i=attribute, m=self._parent_module),
+i=attribute, m=self._parent_module.__name__),
 DeprecationWarning)
 
 loaded_module = self._loaded_modules[module]

[jira] [Created] (AIRFLOW-2470) Retry Time is not honored correctly

[Abhishek Khurana (JIRA)]

Abhishek Khurana created AIRFLOW-2470:
-

 Summary: Retry Time is not honored correctly 
 Key: AIRFLOW-2470
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2470
 Project: Apache Airflow
  Issue Type: Bug
  Components: DagRun
Reporter: Abhishek Khurana


The retry time that is mentioned in the Dag is not honored correctly.

For example, we posted a Dag with retry count=2 and with a delay of 5 seconds 
and following are the observations : 

1) We observed that when there was no load on scheduler, it took about 30 
seconds to retry each time so time of 5 seconds was not being honored here

2)We tried this again where there was existing load on scheduler. In this case, 
it took 25 minutes for each retry. 

 

Expected : It should have retried in 5 seconds which is not the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)