[jira] [Closed] (AIRFLOW-1407) Enable tests in Spark submit operator

2017-07-12 Thread Fokko Driesprong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong closed AIRFLOW-1407.
-
Resolution: Duplicate

> Enable tests in Spark submit operator
> -
>
> Key: AIRFLOW-1407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1407
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>
> Currently the tests are disabled at the spark-submit operator for Python 3. 
> This is definitely something that we don't want. I would like to enable the 
> tests and fix the underlying issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1407) Enable tests in Spark submit operator

2017-07-12 Thread Fokko Driesprong (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084475#comment-16084475
 ] 

Fokko Driesprong commented on AIRFLOW-1407:
---

Ok, missed that one! Good work

> Enable tests in Spark submit operator
> -
>
> Key: AIRFLOW-1407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1407
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>
> Currently the tests are disabled at the spark-submit operator for Python 3. 
> This is definitely something that we don't want. I would like to enable the 
> tests and fix the underlying issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (AIRFLOW-1402) Cleanup SafeConfigParser DeprecationWarning

2017-07-12 Thread Chris Riccomini (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-1402.

   Resolution: Fixed
Fix Version/s: 1.9.0

> Cleanup SafeConfigParser DeprecationWarning
> ---
>
> Key: AIRFLOW-1402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1402
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ron Fung
>Assignee: Ron Fung
>Priority: Minor
> Fix For: 1.9.0
>
>
> {code} DeprecationWarning: The SafeConfigParser class has been renamed to 
> ConfigParser in Python 3.2. This alias will be removed in future versions. 
> Use ConfigParser directly instead. {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1408) Add CreditCards.com

2017-07-12 Thread Vikas Malhotra (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Malhotra updated AIRFLOW-1408:

Description: 
Add to README.md:
```
 
[CreditCards.com](https://www.creditcards.com/)[[[~vikas.malhotra]](https://github.com/vmAggies),
 (https://github.com/jay-wallaby)]
```

  was:
Add to README.md:
```
 [iHeartRadio](http://www.iheart.com/)[[@yiwang](https://github.com/yiwang)]
```


> Add CreditCards.com
> ---
>
> Key: AIRFLOW-1408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1408
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: docs
>Reporter: Vikas Malhotra
>Priority: Trivial
>  Labels: beginner, document, newbie
>
> Add to README.md:
> ```
>  
> [CreditCards.com](https://www.creditcards.com/)[[[~vikas.malhotra]](https://github.com/vmAggies),
>  (https://github.com/jay-wallaby)]
> ```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1402] Cleanup SafeConfigParser DeprecationWarning

2017-07-12 Thread criccomini 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master f5ffb9deb -> 38c86bbbc


[AIRFLOW-1402] Cleanup SafeConfigParser DeprecationWarning

Closes #2435 from ronfung/work


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/38c86bbb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/38c86bbb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/38c86bbb

Branch: refs/heads/master
Commit: 38c86bbbcbc85ec470f31a33f2a480ea45510a45
Parents: f5ffb9d
Author: Ron Fung 
Authored: Wed Jul 12 10:09:24 2017 -0700
Committer: Chris Riccomini 
Committed: Wed Jul 12 10:09:24 2017 -0700

--
 airflow/configuration.py | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/38c86bbb/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index e325067..460d975 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -42,7 +42,10 @@ warnings.filterwarnings(
 warnings.filterwarnings(
 action='default', category=PendingDeprecationWarning, module='airflow')
 
-ConfigParser = configparser.SafeConfigParser
+if six.PY3:
+ConfigParser = configparser.ConfigParser
+else:
+ConfigParser = configparser.SafeConfigParser
 
 
 def generate_fernet_key():

[jira] [Commented] (AIRFLOW-1402) Cleanup SafeConfigParser DeprecationWarning

2017-07-12 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084317#comment-16084317
 ] 

ASF subversion and git services commented on AIRFLOW-1402:
--

Commit 38c86bbbcbc85ec470f31a33f2a480ea45510a45 in incubator-airflow's branch 
refs/heads/master from [~ronfung]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=38c86bb ]

[AIRFLOW-1402] Cleanup SafeConfigParser DeprecationWarning

Closes #2435 from ronfung/work


> Cleanup SafeConfigParser DeprecationWarning
> ---
>
> Key: AIRFLOW-1402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1402
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ron Fung
>Assignee: Ron Fung
>Priority: Minor
>
> {code} DeprecationWarning: The SafeConfigParser class has been renamed to 
> ConfigParser in Python 3.2. This alias will be removed in future versions. 
> Use ConfigParser directly instead. {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1408) Add CreditCards.com

2017-07-12 Thread Vikas Malhotra (JIRA) 
Vikas Malhotra created AIRFLOW-1408:
---

 Summary: Add CreditCards.com
 Key: AIRFLOW-1408
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1408
 Project: Apache Airflow
  Issue Type: Wish
  Components: docs
Reporter: Vikas Malhotra
Priority: Trivial


Add to README.md:
```
 [iHeartRadio](http://www.iheart.com/)[[@yiwang](https://github.com/yiwang)]
```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1407) Enable tests in Spark submit operator

2017-07-12 Thread Ash Berlin-Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084147#comment-16084147
 ] 

Ash Berlin-Taylor commented on AIRFLOW-1407:


Duplicate of AIRFLOW-1393, and I've made a start fixing the tests (py3 binary 
vs string differences in the data tests are feeding)

> Enable tests in Spark submit operator
> -
>
> Key: AIRFLOW-1407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1407
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>
> Currently the tests are disabled at the spark-submit operator for Python 3. 
> This is definitely something that we don't want. I would like to enable the 
> tests and fix the underlying issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1407) Enable tests in Spark submit operator

2017-07-12 Thread Fokko Driesprong (JIRA) 
Fokko Driesprong created AIRFLOW-1407:
-

 Summary: Enable tests in Spark submit operator
 Key: AIRFLOW-1407
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1407
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Fokko Driesprong


Currently the tests are disabled at the spark-submit operator for Python 3. 
This is definitely something that we don't want. I would like to enable the 
tests and fix the underlying issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill

2017-07-12 Thread Tobias Feldhaus (JIRA) 
Tobias Feldhaus created AIRFLOW-1406:


 Summary: ignore_depends_on_past property ignored by backfill
 Key: AIRFLOW-1406
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
 Project: Apache Airflow
  Issue Type: Bug
  Components: backfill
Affects Versions: Airflow 1.8
Reporter: Tobias Feldhaus


Trying to run the following command does not ignore the first depends_on_past 
dependency of the DAG that has depends_on_past set to true:

{code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I 
google_pipelines{code}

neither does this one:

{code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 
--ignore_depends_on_past google_pipelines {code}

both result in:
{code}
BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run 
because of "depends_on_past" relationships. Try running the backfill with the 
option "ignore_first_depends_on_past=True" or passing "-I" at the command line.
{code}
Trying to run with
{code}
--ignore_depends_on_past=True 
{code}

yields:
{code}
airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored 
explicit argument 'True'
{code}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1400) catchup=False caused exception

2017-07-12 Thread Xi Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083708#comment-16083708
 ] 

Xi Wang commented on AIRFLOW-1400:
--

[~dxhuang] 
It's @once. is that the issue? thx.

> catchup=False caused exception
> --
>
> Key: AIRFLOW-1400
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1400
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.8
>Reporter: Xi Wang
>
> When I set of the task with catchup=False, it threw a error as 
> follow(logs/scheduler/my_dag_name):
> [2017-07-10 15:13:12,534] {jobs.py:354} DagFileProcessor373 ERROR - Got an 
> exception! Propagating...
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper
> pickle_dags)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in 
> wrapper
> result = func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in 
> process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
>   File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1171, in 
> _process_dags
> dag_run = self.create_dag_run(dag)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in 
> wrapper
> result = func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 776, in 
> create_dag_run
> if next_start <= now:
> TypeError: can't compare datetime.datetime to NoneType
> It seems next_start was not defined properly, 
> (https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L777)
> Any help is appreciated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1326) Contrib Spark Submit Hook should be able to produce arguments containing spaces.

2017-07-12 Thread Bolke de Bruin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1326.
-
   Resolution: Fixed
Fix Version/s: (was: Airflow 1.8)
   (was: Airflow 2.0)
   1.8.3

Issue resolved by pull request #2382
[https://github.com/apache/incubator-airflow/pull/2382]

> Contrib Spark Submit Hook should be able to produce arguments containing 
> spaces.
> 
>
> Key: AIRFLOW-1326
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1326
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, hooks
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Ash Berlin-Taylor
>Assignee: Vianney FOUCAULT
> Fix For: 1.8.3
>
>
> AIRFLOW-1184 introduced a bug making it impossible to include an argument 
> with an embedded space in the SparkSubmitOperator. (I believe the original 
> bug report should have been closed as wontfix/not-a-bug as it was user error 
> in how the SparkSubmitOperator is being called)
> i.e. the correct way to achieve what was wanted in the bug is like this:
> {code}
> 'application_args': [
> '-f', ' foo',
> '--bar', 'bar',
> '--start', '{{ macros.ds_add(ds, -1)}}',
> '--end', '{{ ds }}'
> ]
> {code}
> This was possibly compounded as the tests were showing something that would 
> never "work"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1184) Contrib Spark Submit Hook does not split argument and argument value

2017-07-12 Thread Bolke de Bruin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1184.
-
   Resolution: Fixed
Fix Version/s: (was: Airflow 1.8)
   (was: Airflow 2.0)
   1.8.3

Issue resolved by pull request #2382
[https://github.com/apache/incubator-airflow/pull/2382]

> Contrib Spark Submit Hook does not split argument and argument value
> 
>
> Key: AIRFLOW-1184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1184
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, hooks
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
> Fix For: 1.8.3
>
>
> Python Popen expect a list as command. Spark submit too, as: 
> * ['--option value'] 
> is not the same as 
> * ['--option', 'value']
> in regards of spark. eg spark logs : (yarn logs)
> Error: Unknown option --end 2017-05-08
> Error: Unknown option --begin 2017-05-07
> Error: Unknown option --db_name mydb
> Error: Missing option --begin
> Error: Missing option --end
> Error: Missing option --db_name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1326) Contrib Spark Submit Hook should be able to produce arguments containing spaces.

2017-07-12 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083694#comment-16083694
 ] 

ASF subversion and git services commented on AIRFLOW-1326:
--

Commit f5ffb9deb51e896cdda8992c8b28102f7d650662 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f5ffb9d ]

[AIRFLOW-1326][[AIRFLOW-1326][AIRFLOW-1184] Don't split argument array -- it's 
already an array.[

Closes #2382 from ashb/spark-submit-operator-
preserve-spaces


> Contrib Spark Submit Hook should be able to produce arguments containing 
> spaces.
> 
>
> Key: AIRFLOW-1326
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1326
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, hooks
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Ash Berlin-Taylor
>Assignee: Vianney FOUCAULT
> Fix For: Airflow 2.0, Airflow 1.8
>
>
> AIRFLOW-1184 introduced a bug making it impossible to include an argument 
> with an embedded space in the SparkSubmitOperator. (I believe the original 
> bug report should have been closed as wontfix/not-a-bug as it was user error 
> in how the SparkSubmitOperator is being called)
> i.e. the correct way to achieve what was wanted in the bug is like this:
> {code}
> 'application_args': [
> '-f', ' foo',
> '--bar', 'bar',
> '--start', '{{ macros.ds_add(ds, -1)}}',
> '--end', '{{ ds }}'
> ]
> {code}
> This was possibly compounded as the tests were showing something that would 
> never "work"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1184) Contrib Spark Submit Hook does not split argument and argument value

2017-07-12 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083696#comment-16083696
 ] 

ASF subversion and git services commented on AIRFLOW-1184:
--

Commit f5ffb9deb51e896cdda8992c8b28102f7d650662 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f5ffb9d ]

[AIRFLOW-1326][[AIRFLOW-1326][AIRFLOW-1184] Don't split argument array -- it's 
already an array.[

Closes #2382 from ashb/spark-submit-operator-
preserve-spaces


> Contrib Spark Submit Hook does not split argument and argument value
> 
>
> Key: AIRFLOW-1184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1184
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, hooks
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
> Fix For: Airflow 2.0, Airflow 1.8
>
>
> Python Popen expect a list as command. Spark submit too, as: 
> * ['--option value'] 
> is not the same as 
> * ['--option', 'value']
> in regards of spark. eg spark logs : (yarn logs)
> Error: Unknown option --end 2017-05-08
> Error: Unknown option --begin 2017-05-07
> Error: Unknown option --db_name mydb
> Error: Missing option --begin
> Error: Missing option --end
> Error: Missing option --db_name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1326) Contrib Spark Submit Hook should be able to produce arguments containing spaces.

2017-07-12 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083695#comment-16083695
 ] 

ASF subversion and git services commented on AIRFLOW-1326:
--

Commit f5ffb9deb51e896cdda8992c8b28102f7d650662 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f5ffb9d ]

[AIRFLOW-1326][[AIRFLOW-1326][AIRFLOW-1184] Don't split argument array -- it's 
already an array.[

Closes #2382 from ashb/spark-submit-operator-
preserve-spaces


> Contrib Spark Submit Hook should be able to produce arguments containing 
> spaces.
> 
>
> Key: AIRFLOW-1326
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1326
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, hooks
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Ash Berlin-Taylor
>Assignee: Vianney FOUCAULT
> Fix For: Airflow 2.0, Airflow 1.8
>
>
> AIRFLOW-1184 introduced a bug making it impossible to include an argument 
> with an embedded space in the SparkSubmitOperator. (I believe the original 
> bug report should have been closed as wontfix/not-a-bug as it was user error 
> in how the SparkSubmitOperator is being called)
> i.e. the correct way to achieve what was wanted in the bug is like this:
> {code}
> 'application_args': [
> '-f', ' foo',
> '--bar', 'bar',
> '--start', '{{ macros.ds_add(ds, -1)}}',
> '--end', '{{ ds }}'
> ]
> {code}
> This was possibly compounded as the tests were showing something that would 
> never "work"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1326][[AIRFLOW-1326][AIRFLOW-1184] Don't split argument array -- it's already an array.[

2017-07-12 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 71594751c -> f5ffb9deb


[AIRFLOW-1326][[AIRFLOW-1326][AIRFLOW-1184] Don't split argument array -- it's 
already an array.[

Closes #2382 from ashb/spark-submit-operator-
preserve-spaces


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f5ffb9de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f5ffb9de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f5ffb9de

Branch: refs/heads/master
Commit: f5ffb9deb51e896cdda8992c8b28102f7d650662
Parents: 7159475
Author: Ash Berlin-Taylor 
Authored: Wed Jul 12 11:27:39 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Jul 12 11:27:39 2017 +0200

--
 airflow/contrib/hooks/spark_submit_hook.py  |  8 ++---
 tests/contrib/hooks/test_spark_submit_hook.py   |  6 ++--
 .../operators/test_spark_submit_operator.py | 31 +++-
 3 files changed, 23 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f5ffb9de/airflow/contrib/hooks/spark_submit_hook.py
--
diff --git a/airflow/contrib/hooks/spark_submit_hook.py 
b/airflow/contrib/hooks/spark_submit_hook.py
index f3d6e34..88d547b 100644
--- a/airflow/contrib/hooks/spark_submit_hook.py
+++ b/airflow/contrib/hooks/spark_submit_hook.py
@@ -194,12 +194,8 @@ class SparkSubmitHook(BaseHook):
 
 # Append any application arguments
 if self._application_args:
-for arg in self._application_args:
-if len(arg.split()) > 1:
-for splitted_option in arg.split():
-connection_cmd += [splitted_option]
-else:
-connection_cmd += [arg]
+connection_cmd += self._application_args
+
 logging.debug("Spark-Submit cmd: {}".format(connection_cmd))
 
 return connection_cmd

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f5ffb9de/tests/contrib/hooks/test_spark_submit_hook.py
--
diff --git a/tests/contrib/hooks/test_spark_submit_hook.py 
b/tests/contrib/hooks/test_spark_submit_hook.py
index ef0857b..98e959b 100644
--- a/tests/contrib/hooks/test_spark_submit_hook.py
+++ b/tests/contrib/hooks/test_spark_submit_hook.py
@@ -45,8 +45,9 @@ class TestSparkSubmitHook(unittest.TestCase):
 'driver_memory': '3g',
 'java_class': 'com.foo.bar.AppMain',
 'application_args': [
-'-f foo',
-'--bar bar',
+'-f', 'foo',
+'--bar', 'bar',
+'--with-spaces', 'args should keep embdedded spaces',
 'baz'
 ]
 }
@@ -129,6 +130,7 @@ class TestSparkSubmitHook(unittest.TestCase):
 'test_application.py',
 '-f', 'foo',
 '--bar', 'bar',
+'--with-spaces', 'args should keep embdedded spaces',
 'baz'
 ]
 self.assertEquals(expected_build_cmd, cmd)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f5ffb9de/tests/contrib/operators/test_spark_submit_operator.py
--
diff --git a/tests/contrib/operators/test_spark_submit_operator.py 
b/tests/contrib/operators/test_spark_submit_operator.py
index 09c5a93..1d41941 100644
--- a/tests/contrib/operators/test_spark_submit_operator.py
+++ b/tests/contrib/operators/test_spark_submit_operator.py
@@ -46,10 +46,11 @@ class TestSparkSubmitOperator(unittest.TestCase):
 'driver_memory': '3g',
 'java_class': 'com.foo.bar.AppMain',
 'application_args': [
-'-f foo',
-'--bar bar',
-'--start {{ macros.ds_add(ds, -1)}}',
-'--end {{ ds }}'
+'-f', 'foo',
+'--bar', 'bar',
+'--start', '{{ macros.ds_add(ds, -1)}}',
+'--end', '{{ ds }}',
+'--with-spaces', 'args should keep embdedded spaces',
 ]
 }
 
@@ -95,10 +96,11 @@ class TestSparkSubmitOperator(unittest.TestCase):
 'driver_memory': '3g',
 'java_class': 'com.foo.bar.AppMain',
 'application_args': [
-'-f foo',
-'--bar bar',
-'--start {{ macros.ds_add(ds, -1)}}',
-'--end {{ ds }}'
+'-f', 'foo',
+'--bar', 'bar',
+'--start', '{{ macros.ds_add(ds, -1)}}',
+'--end', '{{ ds }}',
+'--with-spaces', 'args should keep embdedded spaces',
 ]
 
 }
@@ -130,14 +132,15 @@ class

[jira] [Created] (AIRFLOW-1405) Airflow v 1.8.1 unable to properly initialize with MySQL

2017-07-12 Thread Aakash Bhardwaj (JIRA) 
Aakash Bhardwaj created AIRFLOW-1405:


 Summary: Airflow v 1.8.1 unable to properly initialize with MySQL
 Key: AIRFLOW-1405
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1405
 Project: Apache Airflow
  Issue Type: Bug
  Components: db
Affects Versions: 1.8.1
 Environment: CentOS7
Reporter: Aakash Bhardwaj
 Fix For: 1.8.1
 Attachments: error_log.txt

While working on a CentOS7 system, I was trying to configure Airflow version 
1.8.1 to run with MySql in the backend.
I have installed Airflow in a Virtual Environment, and the MySQL has a database 
named airflow (default).
But on running the command -
{code:shell}
airflow initdb
{code}
the following error is reported

{noformat}
[2017-07-12 13:22:36,558] {__init__.py:57} INFO - Using executor LocalExecutor
DB: mysql://airflow:***@localhost/airflow
[2017-07-12 13:22:37,218] {db.py:287} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, 
Add fractional seconds to mysql tables
Traceback (most recent call last):
  File "/opt/airflow_virtual_environment/airflow_venv/bin/airflow", line 28, in 

args.func(args)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/bin/cli.py",
 line 951, in initdb
db_utils.initdb()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/utils/db.py",
 line 106, in initdb
upgradedb()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/utils/db.py",
 line 294, in upgradedb
command.upgrade(config, 'heads')
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/command.py",
 line 174, in upgrade
script.run_env()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/script/base.py",
 line 416, in run_env
util.load_python_file(self.dir, 'env.py')
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/util/pyfiles.py",
 line 93, in load_python_file
module = load_module_py(module_id, path)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/util/compat.py",
 line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/env.py",
 line 86, in 
run_migrations_online()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/env.py",
 line 81, in run_migrations_online
context.run_migrations()
  File "", line 8, in run_migrations
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/runtime/environment.py",
 line 807, in run_migrations
self.get_context().run_migrations(**kw)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/runtime/migration.py",
 line 321, in run_migrations
step.migration_fn(**kw)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py",
 line 36, in upgrade
op.alter_column(table_name='dag', column_name='last_scheduler_run', 
type_=mysql.DATETIME(fsp=6))
  File "", line 8, in alter_column
  File "", line 3, in alter_column
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/ops.py",
 line 1420, in alter_column
return operations.invoke(alt)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/base.py",
 line 318, in invoke
return fn(self, operation)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/toimpl.py",
 line 53, in alter_column
**operation.kw
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/ddl/mysql.py",
 line 67, in alter_column
else existing_autoincrement
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/ddl/impl.py",
 line 118, in _exec
return conn.execute(construct, *multiparams, **params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 945, in execute
return meth(self, multiparams, params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
 line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1002, in

[jira] [Updated] (AIRFLOW-1405) Airflow v 1.8.1 unable to properly initialize with MySQL

2017-07-12 Thread Aakash Bhardwaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aakash Bhardwaj updated AIRFLOW-1405:
-
Description: 
While working on a CentOS7 system, I was trying to configure Airflow version 
1.8.1 to run with MySql in the backend.
I have installed Airflow in a Virtual Environment, and the MySQL has a database 
named airflow (default).
But on running the command -
{code:none}
airflow initdb
{code}
the following error is reported

{noformat}
[2017-07-12 13:22:36,558] {__init__.py:57} INFO - Using executor LocalExecutor
DB: mysql://airflow:***@localhost/airflow
[2017-07-12 13:22:37,218] {db.py:287} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, 
Add fractional seconds to mysql tables
Traceback (most recent call last):
  File "/opt/airflow_virtual_environment/airflow_venv/bin/airflow", line 28, in 

args.func(args)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/bin/cli.py",
 line 951, in initdb
db_utils.initdb()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/utils/db.py",
 line 106, in initdb
upgradedb()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/utils/db.py",
 line 294, in upgradedb
command.upgrade(config, 'heads')
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/command.py",
 line 174, in upgrade
script.run_env()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/script/base.py",
 line 416, in run_env
util.load_python_file(self.dir, 'env.py')
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/util/pyfiles.py",
 line 93, in load_python_file
module = load_module_py(module_id, path)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/util/compat.py",
 line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/env.py",
 line 86, in 
run_migrations_online()
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/env.py",
 line 81, in run_migrations_online
context.run_migrations()
  File "", line 8, in run_migrations
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/runtime/environment.py",
 line 807, in run_migrations
self.get_context().run_migrations(**kw)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/runtime/migration.py",
 line 321, in run_migrations
step.migration_fn(**kw)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py",
 line 36, in upgrade
op.alter_column(table_name='dag', column_name='last_scheduler_run', 
type_=mysql.DATETIME(fsp=6))
  File "", line 8, in alter_column
  File "", line 3, in alter_column
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/ops.py",
 line 1420, in alter_column
return operations.invoke(alt)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/base.py",
 line 318, in invoke
return fn(self, operation)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/operations/toimpl.py",
 line 53, in alter_column
**operation.kw
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/ddl/mysql.py",
 line 67, in alter_column
else existing_autoincrement
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/alembic/ddl/impl.py",
 line 118, in _exec
return conn.execute(construct, *multiparams, **params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 945, in execute
return meth(self, multiparams, params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
 line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1002, in _execute_ddl
compiled
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1189, in _execute_context
context)
  File 
"/opt/airflow_virtual_environment/airflow_venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py",

[jira] [Created] (AIRFLOW-1404) Big Query Operator add max_billing_tier, create_disposition and flatten_results

2017-07-12 Thread Alex Aslan (JIRA) 
Alex Aslan created AIRFLOW-1404:
---

 Summary: Big Query Operator add max_billing_tier, 
create_disposition and flatten_results 
 Key: AIRFLOW-1404
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1404
 Project: Apache Airflow
  Issue Type: Wish
  Components: operators
Affects Versions: Airflow 1.8
Reporter: Alex Aslan
Priority: Trivial


I would like to have the option to set explicitly the params max_billing_tier, 
flatten_results and create_disposition in the contrib big query operator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)