[jira] [Commented] (AIRFLOW-341) Add Resources To Tasks

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385070#comment-15385070
 ] 

ASF subversion and git services commented on AIRFLOW-341:
-

Commit 348f25f08af2c02627cec04453564edd2fb69fa3 in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=348f25f ]

[AIRFLOW-341][operators] Add resource requirement attributes to operators

This PR adds optional resource requirements for tasks for use with
resource managers such as Yarn and Mesos.

Considerations:
- I chose to force users to encapsulate resources in a resources object
e.g. Resources(cpu=1) instead of just cpu=1 in their dag attributes.
This creates the pain of having to import Resources for almost every
DAG. I think this is kind of important for scoping/namespacing which we
should start doing.
- Once resources are used by executors we need to add documentation for
these new resources (and examples)

Testing Done:
- New/existing unit tests

plypaul artwr mistercrunch jlowin bolkedebruin criccomini

Closes #1669 from aoen/ddavydov/ddavydov/augment_tasks_with_resources


> Add Resources To Tasks
> --
>
> Key: AIRFLOW-341
> URL: https://issues.apache.org/jira/browse/AIRFLOW-341
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>
> Tasks should have optional resource requirements for use with resource 
> managers such as yarn and mesos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-347) Empty dagruns don't appear in tree view

[Dan Davydov (JIRA)]

Dan Davydov created AIRFLOW-347:
---

 Summary: Empty dagruns don't appear in tree view
 Key: AIRFLOW-347
 URL: https://issues.apache.org/jira/browse/AIRFLOW-347
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui, webserver
Reporter: Dan Davydov
Priority: Minor


Dagruns with no task instances don't show up in tree view which can be 
confusing if for example there are three dagruns with the middle one failing 
and not appearing in the UI (it makes it look like the dagrun was skipped).

The tree view in the UI should show all existing dagruns in the date range 
specified by the user, not just those with task instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-346) ECS Hook, Operator, Sensor

[Rob Froetscher (JIRA)]

Rob Froetscher created AIRFLOW-346:
--

 Summary: ECS Hook, Operator, Sensor
 Key: AIRFLOW-346
 URL: https://issues.apache.org/jira/browse/AIRFLOW-346
 Project: Apache Airflow
  Issue Type: New Feature
Reporter: Rob Froetscher
Assignee: Rob Froetscher
Priority: Minor


It would be good to have an ECS operator that descends from the AWS hook.

ECS hook, descending from the AWS hook that allows general interaction with ECS
ECS operator to submit a new task definition
ECS operator to kick off a task
ECS operator to update a service with a new task definition
ECS sensor to check on the status of a task

There is a sort of half done one here 
https://github.com/apache/incubator-airflow/pull/652. But the author says they 
have abandoned it.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-343) Schema setting not propagated in HiveServer2Hook

[Devon Berry (JIRA)]

Devon Berry created AIRFLOW-343:
---

 Summary: Schema setting not propagated in HiveServer2Hook
 Key: AIRFLOW-343
 URL: https://issues.apache.org/jira/browse/AIRFLOW-343
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: Airflow 2.0, Airflow 1.8, Airflow 1.7.1
Reporter: Devon Berry
Assignee: Devon Berry


Although methods like #get_results, #get_records, and #get_pandas_df allow you 
to specify a schema, the underlying delegate #get_conn does not support setting 
the schema:
https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/hive_hooks.py#L576

This story is to fix the wiring so that schema can be overridden.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-261) Add more fields for EmailOperator

[Ajay Yadava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Yadava updated AIRFLOW-261:

External issue URL: https://github.com/apache/incubator-airflow/pull/1670

> Add more fields for EmailOperator
> -
>
> Key: AIRFLOW-261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-261
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Ajay Yadava
>Assignee: Ajay Yadava
>
> It will be nice to be able to specify cc, bcc recipients for the 
> EmailOperator. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-342) exception in 'airflow scheduler' : Connection reset by peer

[Hila Visan (JIRA)]

Hila Visan created AIRFLOW-342:
--

 Summary:  exception in 'airflow scheduler' : Connection reset by 
peer
 Key: AIRFLOW-342
 URL: https://issues.apache.org/jira/browse/AIRFLOW-342
 Project: Apache Airflow
  Issue Type: Bug
  Components: celery, scheduler
Affects Versions: Airflow 1.7.1.3
 Environment: OS: Red Hat Enterprise Linux Server 7.2 (Maipo)
Python: 2.7.5
Airflow: 1.7.1.3
Reporter: Hila Visan
Assignee: Siddharth Anand


'airflow scheduler' command throws an exception when running it. 
Despite the exception, the workers run the tasks from the queues as expected.

Error details:
 
[2016-06-30 19:00:10,130] {jobs.py:758} ERROR - [Errno 104] Connection reset by 
peer
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
  File "/usr/lib/python2.7/site-packages/airflow/executors/base_executor.py", 
line 107, in heartbeat
self.sync()
  File "/usr/lib/python2.7/site-packages/airflow/executors/celery_executor.py", 
line 74, in sync
state = async.state
  File "/usr/lib/python2.7/site-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
  File "/usr/lib/python2.7/site-packages/celery/result.py", line 339, in 
_get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/lib/python2.7/site-packages/celery/backends/amqp.py", line 163, in 
get_task_meta
binding.declare()
  File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 521, in declare
   self.exchange.declare(nowait)
  File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
  File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 615, in 
exchange_declare
self._send_method((40, 10), args)
  File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in 
_send_method
self.channel_id, method_sig, args, content,
  File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in 
write_method
write_frame(1, channel, payload)
  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in 
write_frame
frame_type, channel, size, payload, 0xce,
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
[2016-06-30 19:00:10,131] {jobs.py:759} ERROR - Tachycardia!




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-277) Multiple deletions does not work in Task Instances view if using SQLite backend

[Siddharth Anand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Anand closed AIRFLOW-277.
---
Resolution: Fixed

> Multiple deletions does not work in Task Instances view if using SQLite 
> backend
> ---
>
> Key: AIRFLOW-277
> URL: https://issues.apache.org/jira/browse/AIRFLOW-277
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.1.3
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>
> AIRFLOW-252 fixed the trash button on Task Instances view, but "With 
> selected" > "Delete" menu still seems not working on SQLite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-252) Raise Sqlite exceptions when deleting tasks instance in WebUI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383666#comment-15383666
 ] 

ASF subversion and git services commented on AIRFLOW-252:
-

Commit a88ee53e88f84fd64744c8e3df27caae778e0eba in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a88ee53 ]

[AIRFLOW-277] Multiple deletions does not work in Task Instances view if using 
SQLite backend

AIRFLOW-252 fixed the trash button in "Task Instances" view on Web UI,
but "With selected" > "Delete" menu does still not working for the same reason.
This patch fixes this problem by overriding Flask-Admin's method if the backend 
is SQLite.


> Raise Sqlite exceptions when deleting tasks instance in WebUI
> -
>
> Key: AIRFLOW-252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-252
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.1.3
> Environment: Linux 0f857c51a5ce 4.0.9-boot2docker #1 SMP Thu Sep 10 
> 20:39:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: ryanchou
>Assignee: Kengo Seki
> Fix For: Airflow 1.8
>
>
> When I try to delete a task instances on browse -> tasks instances page 
> through WebUI. It raise a Sqlite exception
> The airflow version is v1.7.1.3 I haven't seen the version choice in jira 
> issue.
> ```
>  File 
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", 
> line 607, in process
> raise TypeError("SQLite DateTime type only accepts Python "
> StatementError: (exceptions.TypeError) SQLite DateTime type only accepts 
> Python datetime and date objects as input. [SQL: u'SELECT 
> task_instance.task_id AS task_instance_task_id, task_in
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-277) Multiple deletions does not work in Task Instances view if using SQLite backend

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383665#comment-15383665
 ] 

ASF subversion and git services commented on AIRFLOW-277:
-

Commit a88ee53e88f84fd64744c8e3df27caae778e0eba in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a88ee53 ]

[AIRFLOW-277] Multiple deletions does not work in Task Instances view if using 
SQLite backend

AIRFLOW-252 fixed the trash button in "Task Instances" view on Web UI,
but "With selected" > "Delete" menu does still not working for the same reason.
This patch fixes this problem by overriding Flask-Admin's method if the backend 
is SQLite.


> Multiple deletions does not work in Task Instances view if using SQLite 
> backend
> ---
>
> Key: AIRFLOW-277
> URL: https://issues.apache.org/jira/browse/AIRFLOW-277
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.1.3
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>
> AIRFLOW-252 fixed the trash button on Task Instances view, but "With 
> selected" > "Delete" menu still seems not working on SQLite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-airflow git commit: [AIRFLOW-277] Multiple deletions does not work in Task Instances view if using SQLite backend

[sanand]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master ab69637be -> 5d90d132a


[AIRFLOW-277] Multiple deletions does not work in Task Instances view if using 
SQLite backend

AIRFLOW-252 fixed the trash button in "Task Instances" view on Web UI,
but "With selected" > "Delete" menu does still not working for the same reason.
This patch fixes this problem by overriding Flask-Admin's method if the backend 
is SQLite.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a88ee53e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a88ee53e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a88ee53e

Branch: refs/heads/master
Commit: a88ee53e88f84fd64744c8e3df27caae778e0eba
Parents: 3b84bcb
Author: Kengo Seki 
Authored: Mon Jun 27 03:33:46 2016 +
Committer: Kengo Seki 
Committed: Tue Jun 28 01:53:31 2016 +

--
 airflow/www/views.py | 34 ++
 1 file changed, 34 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a88ee53e/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 1fb3f91..998926a 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -38,6 +38,7 @@ from flask import (
 from flask_admin import BaseView, expose, AdminIndexView
 from flask_admin.contrib.sqla import ModelView
 from flask_admin.actions import action
+from flask_admin.babel import lazy_gettext
 from flask_admin.tools import iterdecode
 from flask_login import flash
 from flask._compat import PY2
@@ -2078,6 +2079,21 @@ class TaskInstanceModelView(ModelViewOnly):
 def action_set_retry(self, ids):
 self.set_task_instance_state(ids, State.UP_FOR_RETRY)
 
+@action('delete',
+lazy_gettext('Delete'),
+lazy_gettext('Are you sure you want to delete selected records?'))
+def action_delete(self, ids):
+"""
+As a workaround for AIRFLOW-277, this method overrides Flask-Admin's 
ModelView.action_delete().
+
+TODO: this method should be removed once the below bug is fixed on 
Flask-Admin side.
+https://github.com/flask-admin/flask-admin/issues/1226
+"""
+if 'sqlite' in conf.get('core', 'sql_alchemy_conn'):
+self.delete_task_instances(ids)
+else:
+super(TaskInstanceModelView, self).action_delete(ids)
+
 @provide_session
 def set_task_instance_state(self, ids, target_state, session=None):
 try:
@@ -2098,6 +2114,24 @@ class TaskInstanceModelView(ModelViewOnly):
 raise Exception("Ooops")
 flash('Failed to set state', 'error')
 
+@provide_session
+def delete_task_instances(self, ids, session=None):
+try:
+TI = models.TaskInstance
+count = 0
+for id in ids:
+task_id, dag_id, execution_date = id.split(',')
+execution_date = datetime.strptime(execution_date, '%Y-%m-%d 
%H:%M:%S')
+count += session.query(TI).filter(TI.task_id == task_id,
+  TI.dag_id == dag_id,
+  TI.execution_date == 
execution_date).delete()
+session.commit()
+flash("{count} task instances were deleted".format(**locals()))
+except Exception as ex:
+if not self.handle_view_exception(ex):
+raise Exception("Ooops")
+flash('Failed to delete', 'error')
+
 def get_one(self, id):
 """
 As a workaround for AIRFLOW-252, this method overrides Flask-Admin's 
ModelView.get_one().

[2/2] incubator-airflow git commit: Merge branch '1631'

[sanand]

Merge branch '1631'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/5d90d132
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/5d90d132
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/5d90d132

Branch: refs/heads/master
Commit: 5d90d132af4b5a455c6f3bb43817f0e46195cf72
Parents: ab69637 a88ee53
Author: Siddharth Anand 
Authored: Mon Jul 18 23:33:47 2016 -0700
Committer: Siddharth Anand 
Committed: Mon Jul 18 23:33:47 2016 -0700

--
 airflow/www/views.py | 34 ++
 1 file changed, 34 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/5d90d132/airflow/www/views.py
--