[jira] [Assigned] (AIRFLOW-936) Ability to clear/mark success all tasks in a dagrun in the UI

[Dan Davydov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dan Davydov reassigned AIRFLOW-936:
---

Assignee: Dan Davydov

> Ability to clear/mark success all tasks in a dagrun in the UI
> -
>
> Key: AIRFLOW-936
> URL: https://issues.apache.org/jira/browse/AIRFLOW-936
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: webserver
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>
> The same way you can clear/mark success a task instance recursively, you 
> should be able to do for a dagrun circle in the UI in the tree view. Users 
> have been resorting to creating dummy root tasks for their DAGs just to clear 
> the other tasks in a dagrun which is a hack.
> This would require changing the current behavior of clicking a dagrun (from 
> editing the dagrun), to having a popup similar to what you get when you click 
> on a task instance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (AIRFLOW-1222) IRQ : Prod Issue : Alerter Error - "can't convert nil into String"

[Siddharth Anand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Anand closed AIRFLOW-1222.

Resolution: Invalid

> IRQ : Prod Issue : Alerter Error - "can't convert nil into String"
> --
>
> Key: AIRFLOW-1222
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1222
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Siddharth Anand
>Assignee: Stephen Cattaneo
>
> Possibly, the nrt-alerter is not handling Nil subjects!
> Error Message: "can't convert nil into String"
> Stack Trace:
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/message_alert_settings.rb:8:in
>  `sub'
> /us
> r/local/agari/cousteau/production/releases/20170510215139/app/models/message_alert_settings.rb:8:in
>  `alert_subject'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/message_alert.rb:189:in
>  `block in notify_original_recipients!'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/message_alert.rb:181:in
>  `each'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/message_alert.rb:181:in
>  `notify_original_recipients!'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/alert_event.rb:59:in
>  `notify_original_recipients!'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/alert_event.rb:37:in
>  `fire!'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:180:in
>  `block in policy_engine'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:175:in
>  `each'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:175:in
>  `each_with_object'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:175:in
>  `policy_engine'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:108:in
>  `block in process_record'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/bundler/gems/agari-rails-engine-a4906b5ae7ca/lib/util.rb:120:in
>  `robust_operation'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/bundler/gems/agari-rails-engine-a4906b5ae7ca/lib/util/db.rb:47:in
>  `robust_operation'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:100:in
>  `process_record'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:81:in
>  `block in process_records'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:79:in
>  `each'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/jobs/nrt_alerter.rb:79:in
>  `process_records'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/gems/aws-kclrb-1.0.1/lib/aws/kclrb/kcl_process.rb:83:in
>  `dispatch_to_processor'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/gems/aws-kclrb-1.0.1/lib/aws/kclrb/kcl_process.rb:68:in
>  `process_action'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/gems/aws-kclrb-1.0.1/lib/aws/kclrb/kcl_process.rb:44:in
>  `run'
> /usr/local/agari/cousteau/production/releases/20170510215139/app/models/data_stream/kinesis/stream_reader.rb:59:in
>  `run'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/bundler/gems/agari-rails-engine-a4906b5ae7ca/lib/cli_app.rb:253:in
>  `main'
> /usr/local/agari/cousteau/production/shared/bundle/ruby/1.9.1/bundler/gems/agari-rails-engine-a4906b5ae7ca/lib/cli_app.rb:71:in
>  `main'
> /usr/local/agari/cousteau/production/current/bin/nrt-alerter:6:in `'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1214) BigQuery hook fail to convert scientific notation string into int

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014685#comment-16014685
 ] 

Chris Riccomini commented on AIRFLOW-1214:
--

What version of Python are you running? I've never seen BQ return scientific 
notation.

> BigQuery hook fail to convert scientific notation string into int
> -
>
> Key: AIRFLOW-1214
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1214
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp, hooks
>Reporter: Richard Lee
>Assignee: Richard Lee
>
> BigQuery sometimes returns integer result like "8.833167174e+11", which will 
> cause conversion error in Python.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1210) Enable DbApiHook unit tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014634#comment-16014634
 ] 

ASF subversion and git services commented on AIRFLOW-1210:
--

Commit a379cc1bcb19061d2fb42a308d66e6c446e36a51 in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a379cc1 ]

[AIRFLOW-1210] Enable DbApiHook unit tests

Closes #2302 from skudriashev/airflow-1210


> Enable DbApiHook unit tests
> ---
>
> Key: AIRFLOW-1210
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1210
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
>Priority: Minor
> Fix For: Airflow 2.0
>
>
> Currently these tests are not running. Enabling them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-1210] Enable DbApiHook unit tests

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master f9ffbbd91 -> a379cc1bc


[AIRFLOW-1210] Enable DbApiHook unit tests

Closes #2302 from skudriashev/airflow-1210


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a379cc1b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a379cc1b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a379cc1b

Branch: refs/heads/master
Commit: a379cc1bcb19061d2fb42a308d66e6c446e36a51
Parents: f9ffbbd
Author: Stanislav Kudriashev 
Authored: Wed May 17 21:03:44 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 21:03:44 2017 +0200

--
 tests/hooks/__init__.py| 13 +
 tests/hooks/test_dbapi_hook.py |  5 ++---
 2 files changed, 15 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a379cc1b/tests/hooks/__init__.py
--
diff --git a/tests/hooks/__init__.py b/tests/hooks/__init__.py
new file mode 100644
index 000..9d7677a
--- /dev/null
+++ b/tests/hooks/__init__.py
@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a379cc1b/tests/hooks/test_dbapi_hook.py
--
diff --git a/tests/hooks/test_dbapi_hook.py b/tests/hooks/test_dbapi_hook.py
index f9e7b37..c770955 100644
--- a/tests/hooks/test_dbapi_hook.py
+++ b/tests/hooks/test_dbapi_hook.py
@@ -56,8 +56,7 @@ class TestDbApiHook(unittest.TestCase):
 ("world",)]
 
 self.cur.fetchall.return_value = rows
-
-
+
 self.assertEqual(rows, self.db_hook.get_records(statement, parameters))
 
 self.conn.close.assert_called_once()
@@ -73,4 +72,4 @@ class TestDbApiHook(unittest.TestCase):
 
 self.conn.close.assert_called_once()
 self.cur.close.assert_called_once()
-self.cur.execute.assert_called_once_with(statement)
\ No newline at end of file
+self.cur.execute.assert_called_once_with(statement)

[jira] [Commented] (AIRFLOW-1199) Create modal looks like edit modal (edit tab selected)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014626#comment-16014626
 ] 

ASF subversion and git services commented on AIRFLOW-1199:
--

Commit f9ffbbd918d1e1093f0d2823edea48126bf35c65 in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f9ffbbd ]

[AIRFLOW-1199] Fix create modal

Closes #2293 from skudriashev/airflow-1199


> Create modal looks like edit modal (edit tab selected)
> --
>
> Key: AIRFLOW-1199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.8.2
>
> Attachments: imageedit_4_9834649179.jpg
>
>
> !imageedit_4_9834649179.jpg|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1199) Create modal looks like edit modal (edit tab selected)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014627#comment-16014627
 ] 

ASF subversion and git services commented on AIRFLOW-1199:
--

Commit 2e3fc7ecf0311912e9b92487410ceccbd772c5a2 in incubator-airflow's branch 
refs/heads/v1-8-test from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2e3fc7e ]

[AIRFLOW-1199] Fix create modal

Closes #2293 from skudriashev/airflow-1199

(cherry picked from commit f9ffbbd918d1e1093f0d2823edea48126bf35c65)
Signed-off-by: Bolke de Bruin 


> Create modal looks like edit modal (edit tab selected)
> --
>
> Key: AIRFLOW-1199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.8.2
>
> Attachments: imageedit_4_9834649179.jpg
>
>
> !imageedit_4_9834649179.jpg|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-1199) Create modal looks like edit modal (edit tab selected)

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1199.
-
   Resolution: Fixed
Fix Version/s: (was: Airflow 2.0)
   1.8.2

Issue resolved by pull request #2293
[https://github.com/apache/incubator-airflow/pull/2293]

> Create modal looks like edit modal (edit tab selected)
> --
>
> Key: AIRFLOW-1199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.8.2
>
> Attachments: imageedit_4_9834649179.jpg
>
>
> !imageedit_4_9834649179.jpg|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-1199] Fix create modal

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test e2456f07c -> 2e3fc7ecf


[AIRFLOW-1199] Fix create modal

Closes #2293 from skudriashev/airflow-1199

(cherry picked from commit f9ffbbd918d1e1093f0d2823edea48126bf35c65)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2e3fc7ec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2e3fc7ec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2e3fc7ec

Branch: refs/heads/v1-8-test
Commit: 2e3fc7ecf0311912e9b92487410ceccbd772c5a2
Parents: e2456f0
Author: Stanislav Kudriashev 
Authored: Wed May 17 20:55:19 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:55:33 2017 +0200

--
 airflow/www/templates/airflow/model_create.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2e3fc7ec/airflow/www/templates/airflow/model_create.html
--
diff --git a/airflow/www/templates/airflow/model_create.html 
b/airflow/www/templates/airflow/model_create.html
index bbccb94..017fa2d 100644
--- a/airflow/www/templates/airflow/model_create.html
+++ b/airflow/www/templates/airflow/model_create.html
@@ -15,7 +15,7 @@
   limitations under the License.
 
 #}
-{% extends 'admin/model/edit.html' %}
+{% extends 'admin/model/create.html' %}
 
 {% block body %}
   {% if admin_view.verbose_name %}

incubator-airflow git commit: [AIRFLOW-1199] Fix create modal

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 3acfa048a -> f9ffbbd91


[AIRFLOW-1199] Fix create modal

Closes #2293 from skudriashev/airflow-1199


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f9ffbbd9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f9ffbbd9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f9ffbbd9

Branch: refs/heads/master
Commit: f9ffbbd918d1e1093f0d2823edea48126bf35c65
Parents: 3acfa04
Author: Stanislav Kudriashev 
Authored: Wed May 17 20:55:19 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:55:19 2017 +0200

--
 airflow/www/templates/airflow/model_create.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f9ffbbd9/airflow/www/templates/airflow/model_create.html
--
diff --git a/airflow/www/templates/airflow/model_create.html 
b/airflow/www/templates/airflow/model_create.html
index bbccb94..017fa2d 100644
--- a/airflow/www/templates/airflow/model_create.html
+++ b/airflow/www/templates/airflow/model_create.html
@@ -15,7 +15,7 @@
   limitations under the License.
 
 #}
-{% extends 'admin/model/edit.html' %}
+{% extends 'admin/model/create.html' %}
 
 {% block body %}
   {% if admin_view.verbose_name %}

[jira] [Resolved] (AIRFLOW-1200) Forbid creation of a variable with an empty key

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1200.
-
   Resolution: Fixed
Fix Version/s: (was: Airflow 2.0)
   1.8.2

Issue resolved by pull request #2299
[https://github.com/apache/incubator-airflow/pull/2299]

> Forbid creation of a variable with an empty key
> ---
>
> Key: AIRFLOW-1200
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1200
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.8.2
>
>
> This leads to the following exception:
> {code}
> Traceback (most recent call last):
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 2000, in __call__
> return self.wsgi_app(environ, start_response)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1988, in wsgi_app
> response = self.full_dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1639, in full_dispatch_request
> rv = self.dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 69, in inner
> return self._run_view(f, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 368, in _run_view
> return fn(self, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/model/base.py",
>  line 1900, in index_view
> return_url=self._get_list_url(view_args),
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 308, in render
> return render_template(template, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 134, in render_template
> context, ctx.app)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 116, in _render
> rv = template.render(context)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 989, in render
> return self.environment.handle_exception(exc_info, True)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 754, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/airflow/variable_list.html",
>  line 18, in top-level template code
> {% extends 'admin/model/list.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 6, in top-level template code
> {% import 'admin/model/row_actions.html' as row_actions with context %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/admin/master.html",
>  line 18, in top-level template code
> {% extends 'admin/base.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
>  line 30, in top-level template code
> {% block page_body %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/admin/master.html",
>  line 104, in block "page_body"
> {% block body %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 62, in block 

[jira] [Commented] (AIRFLOW-1200) Forbid creation of a variable with an empty key

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014624#comment-16014624
 ] 

ASF subversion and git services commented on AIRFLOW-1200:
--

Commit e2456f07c8da00c3daef5e6a2a56611ccfc48cf9 in incubator-airflow's branch 
refs/heads/v1-8-test from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e2456f0 ]

[AIRFLOW-1200] Forbid creation of a variable with an empty key

Closes #2299 from skudriashev/airflow-1200

(cherry picked from commit 3acfa048a88342ca059afd9329e7ba4cf1af0929)
Signed-off-by: Bolke de Bruin 


> Forbid creation of a variable with an empty key
> ---
>
> Key: AIRFLOW-1200
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1200
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.8.2
>
>
> This leads to the following exception:
> {code}
> Traceback (most recent call last):
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 2000, in __call__
> return self.wsgi_app(environ, start_response)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1988, in wsgi_app
> response = self.full_dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1639, in full_dispatch_request
> rv = self.dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 69, in inner
> return self._run_view(f, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 368, in _run_view
> return fn(self, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/model/base.py",
>  line 1900, in index_view
> return_url=self._get_list_url(view_args),
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 308, in render
> return render_template(template, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 134, in render_template
> context, ctx.app)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 116, in _render
> rv = template.render(context)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 989, in render
> return self.environment.handle_exception(exc_info, True)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 754, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/airflow/variable_list.html",
>  line 18, in top-level template code
> {% extends 'admin/model/list.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 6, in top-level template code
> {% import 'admin/model/row_actions.html' as row_actions with context %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/admin/master.html",
>  line 18, in top-level template code
> {% extends 'admin/base.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
>  line 30, in top-level template code
> {% block page_body %}
>   

incubator-airflow git commit: [AIRFLOW-1200] Forbid creation of a variable with an empty key

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test 17233ea77 -> e2456f07c


[AIRFLOW-1200] Forbid creation of a variable with an empty key

Closes #2299 from skudriashev/airflow-1200

(cherry picked from commit 3acfa048a88342ca059afd9329e7ba4cf1af0929)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e2456f07
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e2456f07
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e2456f07

Branch: refs/heads/v1-8-test
Commit: e2456f07c8da00c3daef5e6a2a56611ccfc48cf9
Parents: 17233ea
Author: Stanislav Kudriashev 
Authored: Wed May 17 20:53:50 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:54:02 2017 +0200

--
 airflow/www/views.py | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e2456f07/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 3ed58a8..7267ad7 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2138,13 +2138,20 @@ class VariableView(wwwutils.DataProfilingMixin, 
AirflowModelView):
 'rows': 20,
 }
 }
+form_args = {
+'key': {
+'validators': {
+validators.DataRequired(),
+},
+},
+}
 column_sortable_list = (
 'key',
 'val',
 'is_encrypted',
 )
 column_formatters = {
-'val': hidden_field_formatter
+'val': hidden_field_formatter,
 }
 
 # Default flask-admin export functionality doesn't handle serialized json

[jira] [Commented] (AIRFLOW-1200) Forbid creation of a variable with an empty key

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014623#comment-16014623
 ] 

ASF subversion and git services commented on AIRFLOW-1200:
--

Commit 3acfa048a88342ca059afd9329e7ba4cf1af0929 in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3acfa04 ]

[AIRFLOW-1200] Forbid creation of a variable with an empty key

Closes #2299 from skudriashev/airflow-1200


> Forbid creation of a variable with an empty key
> ---
>
> Key: AIRFLOW-1200
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1200
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: Airflow 2.0
>
>
> This leads to the following exception:
> {code}
> Traceback (most recent call last):
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 2000, in __call__
> return self.wsgi_app(environ, start_response)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1988, in wsgi_app
> response = self.full_dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1639, in full_dispatch_request
> rv = self.dispatch_request()
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/app.py",
>  line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 69, in inner
> return self._run_view(f, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 368, in _run_view
> return fn(self, *args, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/model/base.py",
>  line 1900, in index_view
> return_url=self._get_list_url(view_args),
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/base.py",
>  line 308, in render
> return render_template(template, **kwargs)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 134, in render_template
> context, ctx.app)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask/templating.py",
>  line 116, in _render
> rv = template.render(context)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 989, in render
> return self.environment.handle_exception(exc_info, True)
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/jinja2/environment.py",
>  line 754, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/airflow/variable_list.html",
>  line 18, in top-level template code
> {% extends 'admin/model/list.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 6, in top-level template code
> {% import 'admin/model/row_actions.html' as row_actions with context %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/admin/master.html",
>  line 18, in top-level template code
> {% extends 'admin/base.html' %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/.venv/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
>  line 30, in top-level template code
> {% block page_body %}
>   File 
> "/home/skudriashev/Work/incubator-airflow/airflow/www/templates/admin/master.html",
>  line 104, in block 

[jira] [Updated] (AIRFLOW-1207) Enable utils.helpers unit tests

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated AIRFLOW-1207:

Affects Version/s: (was: Airflow 2.0)
   1.8.1

> Enable utils.helpers unit tests
> ---
>
> Key: AIRFLOW-1207
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1207
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Affects Versions: 1.8.1
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.9.0
>
>
> Currently these tests are not running.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-1200] Forbid creation of a variable with an empty key

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 8ac87b2e2 -> 3acfa048a


[AIRFLOW-1200] Forbid creation of a variable with an empty key

Closes #2299 from skudriashev/airflow-1200


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3acfa048
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3acfa048
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3acfa048

Branch: refs/heads/master
Commit: 3acfa048a88342ca059afd9329e7ba4cf1af0929
Parents: 8ac87b2
Author: Stanislav Kudriashev 
Authored: Wed May 17 20:53:50 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:53:50 2017 +0200

--
 airflow/www/views.py | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3acfa048/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 5d52d5e..69f6c04 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2172,13 +2172,20 @@ class VariableView(wwwutils.DataProfilingMixin, 
AirflowModelView):
 'rows': 20,
 }
 }
+form_args = {
+'key': {
+'validators': {
+validators.DataRequired(),
+},
+},
+}
 column_sortable_list = (
 'key',
 'val',
 'is_encrypted',
 )
 column_formatters = {
-'val': hidden_field_formatter
+'val': hidden_field_formatter,
 }
 
 # Default flask-admin export functionality doesn't handle serialized json

[jira] [Commented] (AIRFLOW-1207) Enable utils.helpers unit tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014620#comment-16014620
 ] 

ASF subversion and git services commented on AIRFLOW-1207:
--

Commit 8ac87b2e284ee710c54085bb418976ccc5f7ea9a in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8ac87b2 ]

[AIRFLOW-1207] Enable utils.helpers unit tests

Closes #2300 from skudriashev/airflow-1207


> Enable utils.helpers unit tests
> ---
>
> Key: AIRFLOW-1207
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1207
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.9.0
>
>
> Currently these tests are not running.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-1207) Enable utils.helpers unit tests

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1207.
-
   Resolution: Fixed
Fix Version/s: (was: Airflow 2.0)
   1.9.0

> Enable utils.helpers unit tests
> ---
>
> Key: AIRFLOW-1207
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1207
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.9.0
>
>
> Currently these tests are not running.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-1207] Enable utils.helpers unit tests

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 857850509 -> 8ac87b2e2


[AIRFLOW-1207] Enable utils.helpers unit tests

Closes #2300 from skudriashev/airflow-1207


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8ac87b2e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8ac87b2e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8ac87b2e

Branch: refs/heads/master
Commit: 8ac87b2e284ee710c54085bb418976ccc5f7ea9a
Parents: 8578505
Author: Stanislav Kudriashev 
Authored: Wed May 17 20:52:18 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:52:18 2017 +0200

--
 tests/utils/__init__.py |   3 --
 tests/utils/compression.py  |  97 -
 tests/utils/dates.py|  45 
 tests/utils/helpers.py  |  84 -
 tests/utils/test_compression.py | 100 +++
 tests/utils/test_dates.py   |  46 
 tests/utils/test_helpers.py |  90 +++
 7 files changed, 236 insertions(+), 229 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8ac87b2e/tests/utils/__init__.py
--
diff --git a/tests/utils/__init__.py b/tests/utils/__init__.py
index 6b15998..9d7677a 100644
--- a/tests/utils/__init__.py
+++ b/tests/utils/__init__.py
@@ -11,6 +11,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from .compression import *
-from .dates import *

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8ac87b2e/tests/utils/compression.py
--
diff --git a/tests/utils/compression.py b/tests/utils/compression.py
deleted file mode 100644
index f8e0ebb..000
--- a/tests/utils/compression.py
+++ /dev/null
@@ -1,97 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from airflow.utils import compression
-import unittest
-from tempfile import NamedTemporaryFile, mkdtemp
-import bz2
-import gzip
-import shutil
-import logging
-import errno
-import filecmp
-
-
-class Compression(unittest.TestCase):
-
-def setUp(self):
-self.fn = {}
-try:
-header = "Sno\tSome,Text \n".encode()
-line1 = "1\tAirflow Test\n".encode()
-line2 = "2\tCompressionUtil\n".encode()
-self.tmp_dir = mkdtemp(prefix='test_utils_compression_')
-# create sample txt, gz and bz2 files
-with NamedTemporaryFile(mode='wb+',
-dir=self.tmp_dir,
-delete=False) as f_txt:
-self._set_fn(f_txt.name, '.txt')
-f_txt.writelines([header, line1, line2])
-fn_gz = self._get_fn('.txt') + ".gz"
-with gzip.GzipFile(filename=fn_gz,
-   mode="wb") as f_gz:
-self._set_fn(fn_gz, '.gz')
-f_gz.writelines([header, line1, line2])
-fn_bz2 = self._get_fn('.txt') + '.bz2'
-with bz2.BZ2File(filename=fn_bz2,
- mode="wb") as f_bz2:
-self._set_fn(fn_bz2, '.bz2')
-f_bz2.writelines([header, line1, line2])
-# Base Exception so it catches Keyboard Interrupt
-except BaseException as e:
-logging.error(e)
-self.tearDown()
-
-def tearDown(self):
-try:
-shutil.rmtree(self.tmp_dir)
-except OSError as e:
-# ENOENT - no such file or directory
-if e.errno != errno.ENOENT:
-raise e
-
-# Helper method to create a dictionary of file names and
-# file extension
-def _set_fn(self, fn, ext):
-self.fn[ext] = fn
-
-# Helper method to fetch a file of a
-# certain extension
-def _get_fn(self, ext):
-return self.fn[ext]
-
-def test_uncompress_file(self):
-# Testing txt file type
-

[jira] [Resolved] (AIRFLOW-1213) Add hcatalog parameters to the sqoop operator/hook

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1213.
-
   Resolution: Fixed
Fix Version/s: 1.8.2

> Add hcatalog parameters to the sqoop operator/hook
> --
>
> Key: AIRFLOW-1213
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1213
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
> Fix For: 1.8.2
>
>
> Hi,
> For the Sqoop hook and operator we would like to add the hcatalog parameters 
> --hcatalog-database and --hcatalog-table. These are added in Sqoop 1.4.6:
> https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
> This allows the user to import/export Hive tables instead of hdfs directories.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1213) Add hcatalog parameters to the sqoop operator/hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014613#comment-16014613
 ] 

ASF subversion and git services commented on AIRFLOW-1213:
--

Commit 8578505098521b991c41285e2d81b5be1ded81da in incubator-airflow's branch 
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8578505 ]

[AIRFLOW-1213] Add hcatalog parameters to sqoop

Add parameters to specify a hive table instead of
a hdfs directory.
This will also source the schema from the hive
metastore.

Closes #2305 from Fokko/AIRFLOW-1213-add-hcatalog-
parameters


> Add hcatalog parameters to the sqoop operator/hook
> --
>
> Key: AIRFLOW-1213
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1213
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
> Fix For: 1.8.2
>
>
> Hi,
> For the Sqoop hook and operator we would like to add the hcatalog parameters 
> --hcatalog-database and --hcatalog-table. These are added in Sqoop 1.4.6:
> https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
> This allows the user to import/export Hive tables instead of hdfs directories.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1213) Add hcatalog parameters to the sqoop operator/hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014612#comment-16014612
 ] 

ASF subversion and git services commented on AIRFLOW-1213:
--

Commit 8578505098521b991c41285e2d81b5be1ded81da in incubator-airflow's branch 
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8578505 ]

[AIRFLOW-1213] Add hcatalog parameters to sqoop

Add parameters to specify a hive table instead of
a hdfs directory.
This will also source the schema from the hive
metastore.

Closes #2305 from Fokko/AIRFLOW-1213-add-hcatalog-
parameters


> Add hcatalog parameters to the sqoop operator/hook
> --
>
> Key: AIRFLOW-1213
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1213
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>
> Hi,
> For the Sqoop hook and operator we would like to add the hcatalog parameters 
> --hcatalog-database and --hcatalog-table. These are added in Sqoop 1.4.6:
> https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
> This allows the user to import/export Hive tables instead of hdfs directories.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-1213] Add hcatalog parameters to sqoop

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master d2d3e49ca -> 857850509


[AIRFLOW-1213] Add hcatalog parameters to sqoop

Add parameters to specify a hive table instead of
a hdfs directory.
This will also source the schema from the hive
metastore.

Closes #2305 from Fokko/AIRFLOW-1213-add-hcatalog-
parameters


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/85785050
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/85785050
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/85785050

Branch: refs/heads/master
Commit: 8578505098521b991c41285e2d81b5be1ded81da
Parents: d2d3e49
Author: Fokko Driesprong 
Authored: Wed May 17 20:48:52 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed May 17 20:48:52 2017 +0200

--
 airflow/contrib/hooks/sqoop_hook.py| 15 ---
 airflow/contrib/operators/sqoop_operator.py|  8 
 tests/contrib/hooks/test_sqoop_hook.py |  7 ++-
 tests/contrib/operators/test_sqoop_operator.py |  4 
 4 files changed, 30 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/85785050/airflow/contrib/hooks/sqoop_hook.py
--
diff --git a/airflow/contrib/hooks/sqoop_hook.py 
b/airflow/contrib/hooks/sqoop_hook.py
index 6c5ee58..2dd33ae 100644
--- a/airflow/contrib/hooks/sqoop_hook.py
+++ b/airflow/contrib/hooks/sqoop_hook.py
@@ -52,7 +52,8 @@ class SqoopHook(BaseHook):
 """
 
 def __init__(self, conn_id='sqoop_default', verbose=False,
- num_mappers=None, properties=None):
+ num_mappers=None, hcatalog_database=None,
+ hcatalog_table=None, properties=None):
 # No mutable types in the default parameters
 if properties is None:
 properties = {}
@@ -64,6 +65,8 @@ class SqoopHook(BaseHook):
 self.files = connection_parameters.get('files', None)
 self.archives = connection_parameters.get('archives', None)
 self.password_file = connection_parameters.get('password_file', None)
+self.hcatalog_database = hcatalog_database
+self.hcatalog_table = hcatalog_table
 self.verbose = verbose
 self.num_mappers = str(num_mappers)
 self.properties = properties
@@ -118,6 +121,10 @@ class SqoopHook(BaseHook):
 connection_cmd += ["-archives", self.archives]
 if self.num_mappers:
 connection_cmd += ["--num-mappers", self.num_mappers]
+if self.hcatalog_database:
+connection_cmd += ["--hcatalog-database", self.hcatalog_database]
+if self.hcatalog_table:
+connection_cmd += ["--hcatalog-table", self.hcatalog_table]
 
 for key, value in self.properties.items():
 connection_cmd += ["-D", "{}={}".format(key, value)]
@@ -257,8 +264,10 @@ class SqoopHook(BaseHook):
 if relaxed_isolation:
 cmd += ["--relaxed-isolation"]
 
-# The required options
-cmd += ["--export-dir", export_dir]
+if export_dir:
+cmd += ["--export-dir", export_dir]
+
+# The required option
 cmd += ["--table", table]
 
 return cmd

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/85785050/airflow/contrib/operators/sqoop_operator.py
--
diff --git a/airflow/contrib/operators/sqoop_operator.py 
b/airflow/contrib/operators/sqoop_operator.py
index b0dc88f..c3da176 100644
--- a/airflow/contrib/operators/sqoop_operator.py
+++ b/airflow/contrib/operators/sqoop_operator.py
@@ -57,6 +57,8 @@ class SqoopOperator(BaseOperator):
  verbose=False,
  relaxed_isolation=False,
  properties=None,
+ hcatalog_database=None,
+ hcatalog_table=None,
  *args,
  **kwargs):
 """
@@ -91,6 +93,8 @@ class SqoopOperator(BaseOperator):
 :param driver: Manually specify JDBC driver class to use
 :param verbose: Switch to more verbose logging for debug purposes
 :param relaxed_isolation: use read uncommitted isolation level
+:param hcatalog_database: Specifies the database name for the HCatalog 
table
+:param hcatalog_table: The argument value for this option is the 
HCatalog table
 :param properties: additional JVM properties passed to sqoop
 """
 super(SqoopOperator, self).__init__(*args, **kwargs)
@@ -120,6 +124,8 @@ class SqoopOperator(BaseOperator):
 self.driver = driver
 self.verbose = verbose
 

[jira] [Created] (AIRFLOW-1221) Fix DatabricksSubmitRunOperator Templating

[Andrew Chen (JIRA)]

Andrew Chen created AIRFLOW-1221:


 Summary: Fix DatabricksSubmitRunOperator Templating
 Key: AIRFLOW-1221
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1221
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Andrew Chen


https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/databricks_operator.py#L207

Bug where we mixed up index and content from enumerate.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (AIRFLOW-1220) Can't clear airflow jobs when using schedule_interval of None

[sam sen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-1220:
-
Description: 
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 

{code:none}
[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag stats 
for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType
{code}

  was:
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 

{code:txt}
[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag stats 
for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType
{code}


> Can't clear airflow jobs when using schedule_interval of None
> -
>
> Key: AIRFLOW-1220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1220
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam 

[jira] [Updated] (AIRFLOW-1220) Can't clear airflow jobs when using schedule_interval of None

[sam sen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-1220:
-
Description: 
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 

{code:txt}
[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag stats 
for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType
{code}

  was:
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 


[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag stats 
for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType



> Can't clear airflow jobs when using schedule_interval of None
> -
>
> Key: AIRFLOW-1220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1220
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam sen
>
> My dag 

[jira] [Updated] (AIRFLOW-1220) Can't clear airflow jobs when using schedule_interval of None

[sam sen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-1220:
-
Description: 
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 


[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag stats 
for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType


  was:
My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 

```[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag 
stats for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType
```


> Can't clear airflow jobs when using schedule_interval of None
> -
>
> Key: AIRFLOW-1220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1220
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam sen
>
> My dag is set to 

[jira] [Created] (AIRFLOW-1220) Can't clear airflow jobs when using schedule_interval of None

[sam sen (JIRA)]

sam sen created AIRFLOW-1220:


 Summary: Can't clear airflow jobs when using schedule_interval of 
None
 Key: AIRFLOW-1220
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1220
 Project: Apache Airflow
  Issue Type: Bug
Reporter: sam sen


My dag is set to use `schedule_interval=None`. I'm attempting to clear jobs so 
I can re-run failed jobs but I'm encountering the following errors. 

```[2017-05-17 14:02:14,718] {models.py:3834} WARNING - Could not update dag 
stats for s3_dag_test
[2017-05-17 14:02:14,718] {models.py:3835} ERROR - can't compare 
datetime.datetime to NoneType
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 3831, in 
set_dirty
session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
874, in commit
self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
461, in commit
self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
441, in _prepare_impl
self.session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2139, in flush
self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2259, in _flush
transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2223, in _flush
flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
389, in execute
rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
577, in execute
uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
243, in delete_obj
uowtransaction))
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
357, in _organize_states_for_delete
states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1108, in _connections_for_states
for state in _sort_states(states):
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 
1130, in _sort_states
sorted(persistent, key=lambda q: q.key[1])
TypeError: can't compare datetime.datetime to NoneType
```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-1219) Spark submit hook fix realtime logging.

[Kris Geusebroek (JIRA)]

Kris Geusebroek created AIRFLOW-1219:


 Summary: Spark submit hook fix realtime logging.
 Key: AIRFLOW-1219
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1219
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Kris Geusebroek


Redirecting the logging to airflow doesn't work correctly.

Spark's logging goes to stderr and output of the spark program goes to stdout. 
In the loop of the two sources stdout is used first.
It seems to be blocking until output is written at the end of the spark-submit 
execution.

Fix this by connecting stderr to stdout in the subprocess invocation and only 
pipe stdout to the airflow logging



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-1218) Enable logging in Sqoop hook

[Fokko Driesprong (JIRA)]

Fokko Driesprong created AIRFLOW-1218:
-

 Summary: Enable logging in Sqoop hook
 Key: AIRFLOW-1218
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1218
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Fokko Driesprong


Currently the sqoop hook does not give any output which makes debugging hard, 
therefore we would like to have proper logging from the subprocess to airflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-1217) Enable logging in Sqoop hook

[Fokko Driesprong (JIRA)]

Fokko Driesprong created AIRFLOW-1217:
-

 Summary: Enable logging in Sqoop hook
 Key: AIRFLOW-1217
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1217
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Fokko Driesprong


Currently the sqoop hook does not give any output in Airflow which makes 
debugging hard. We would like to change this and have proper logging from the 
subprocess to airflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-1214) BigQuery hook fail to convert scientific notation string into int

[Richard Lee (JIRA)]

Richard Lee created AIRFLOW-1214:


 Summary: BigQuery hook fail to convert scientific notation string 
into int
 Key: AIRFLOW-1214
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1214
 Project: Apache Airflow
  Issue Type: Bug
  Components: gcp, hooks
Reporter: Richard Lee
Assignee: Richard Lee


BigQuery sometimes returns integer result like "8.833167174e+11", which will 
cause conversion error in Python.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (AIRFLOW-216) Sqoop hook and operator required

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin closed AIRFLOW-216.
--
   Resolution: Fixed
Fix Version/s: 1.8.0

> Sqoop hook and operator required
> 
>
> Key: AIRFLOW-216
> URL: https://issues.apache.org/jira/browse/AIRFLOW-216
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks, operators
>Affects Versions: Airflow 1.7.0
>Reporter: Johan
>Priority: Minor
> Fix For: 1.8.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> It would be nice to have a Sqoop hook and operator for Sqoop 1 which will 
> execute, import, and export Sqoop commands.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (AIRFLOW-105) SqoopHook

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin closed AIRFLOW-105.
--
   Resolution: Fixed
Fix Version/s: 1.8.0

> SqoopHook
> -
>
> Key: AIRFLOW-105
> URL: https://issues.apache.org/jira/browse/AIRFLOW-105
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks
>Affects Versions: Airflow 1.7.1
>Reporter: Bolke de Bruin
> Fix For: 1.8.0
>
>
> Sqoop 1 Hook



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-1213) Add hcatalog parameters to the sqoop operator/hook

[Fokko Driesprong (JIRA)]

Fokko Driesprong created AIRFLOW-1213:
-

 Summary: Add hcatalog parameters to the sqoop operator/hook
 Key: AIRFLOW-1213
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1213
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Fokko Driesprong


Hi,

For the Sqoop hook and operator we would like to add the hcatalog parameters 
--hcatalog-database and --hcatalog-table. These are added in Sqoop 1.4.6:
https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html

This allows the user to import/export Hive tables instead of hdfs directories.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)