[jira] [Commented] (AIRFLOW-563) Dag stuck in "Filling up the DagBag from .." state

[Mohammad (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824889#comment-15824889
 ] 

Mohammad commented on AIRFLOW-563:
--

Is there any solution for this  situation. We are facing the same situation. 
The Dags were running fine until recently and they stopped to run and the log 
shows the only this message and fails after timeout.

[2017-01-16 06:00:43,789] {models.py:154} INFO - Filling up the DagBag

> Dag stuck in "Filling up the DagBag from .." state
> --
>
> Key: AIRFLOW-563
> URL: https://issues.apache.org/jira/browse/AIRFLOW-563
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Vincent Martinez
>Assignee: Vincent Martinez
>Priority: Critical
> Attachments: 2016-10-10T11_00_00.txt, dagrun.png, load_dims.py
>
>
> Hello,
> I have scheduled a dag hourly but the task doesn't run at all, it stays at 
> "Filling up the DagBag from .." step and each hour I have one more Dag id in 
> running state (stuck aswell). Do you have any idea where does the problem 
> comes from ?
> You'll find attached screens, log of the dag for 1 run and the dag file.
> If you need anything else feel free to ask.
> Thanks
> Best regards,
> Vincent



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

svn commit: r17827 - in /dev/incubator/airflow: airflow-1.8.0b1+apache.incubating.tar.gz airflow-1.8.0b1+apache.incubating.tar.gz.asc airflow-1.8.0b1+apache.incubating.tar.gz.sha

[bolke]

Author: bolke
Date: Mon Jan 16 21:46:38 2017
New Revision: 17827

Log:
Add airflow-1.8.0b1+apache.incubating.tar.gz

Added:
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz   (with 
props)
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.sha

Added: dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz
==
Binary file - no diff available.

Propchange: dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.asc
==
--- dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.asc (added)
+++ dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.asc Mon Jan 
16 21:46:38 2017
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIcBAABCgAGBQJYfT7LAAoJEDUZC4PZBaC6assP/1v4rd6kEnvNhhRPfswhUmYR
+662F8W4h5jToxGM/gCs1S9Uyjq/ryQsoyX7rpiiThq73DqCNWOAan1w8mAbu5kVa
+qbjF0ude6bStiZw8y0/shlmeXvdzN3H3d8fLrX9MGcq0JugAusfqYliszb5DIGPN
+p4jyXtoCQMd4TQh72NyqEzk/ba9P16H9oz8ngRJetqubHYl6dpvJ9xJIFe+SVoFl
+205E+LJd6WN+4aq3I7gWdid/L6m/whBarONeu4rg1xHSZeaVMxEzzATvfzGnarYP
+BRJFf2+Qrtlfz24nsn/jIgCjFbxYp+AqdG+GAZYwqUY/elNDxQQwTgY94rqFdRMm
+JZcCu3x4WVDbBchbXovk5mCBC7A68jBmiue61lWkK/SbAOztXX8q109eKLytMk9r
+sjQ66x0Thj9qGVsPRQ62YUIE/RHzT2ActNxLFNqUU8C+q2psVPJUNRH9AO+leXIa
+9NTNny++r7Gg6WoTZkA4ghEdQqD+u+qfPYIgFZclSB93Q2m9Sa9KRsd/Txpz+XJA
+suDYIyC5oRMdnfNkYx4rgBwgX5rs6HfuilFQLQSGhsLvgQKpZknVToIU6aldc5bN
+gF0BJ1h1vQU4lDs5gQb5zqpA0/TLwApgtW8KnSvNXFxaR9PBoAT0Pn2KJyXHo68M
+ryV//T5gKw0HsB58mZxI
+=Ccmz
+-END PGP SIGNATURE-

Added: dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.sha
==
--- dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.sha (added)
+++ dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.sha Mon Jan 
16 21:46:38 2017
@@ -0,0 +1,3 @@
+airflow-1.8.0b1+apache.incubating.tar.gz: 
+C58FE988 2C2CE74C 9D9B625F F62FE9D7 2DD6DC3B AA22D135 35055FB4 CB60A06D 
215A54BB
+ 2D2BB286 9E60FC1C F13D742C 1A6E5703 099EC81F 3CB515D0 301A9460

incubator-airflow git commit: Add incubating specifier to version

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test a46f114a5 -> 8f9a466de


Add incubating specifier to version


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8f9a466d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8f9a466d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8f9a466d

Branch: refs/heads/v1-8-test
Commit: 8f9a466dee5cd9bcd794e95f3d6cc22103dc4fb7
Parents: a46f114
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:42:01 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:42:01 2017 +0100

--
 airflow/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8f9a466d/airflow/version.py
--
diff --git a/airflow/version.py b/airflow/version.py
index 3d3b0c3..0b45eae 100644
--- a/airflow/version.py
+++ b/airflow/version.py
@@ -13,4 +13,4 @@
 # limitations under the License.
 #
 
-version = '1.8.0b1'
+version = '1.8.0b1+apache.incubating'

[jira] [Commented] (AIRFLOW-757) Set sensible location for processor logs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824601#comment-15824601
 ] 

ASF subversion and git services commented on AIRFLOW-757:
-

Commit a9ab119b4081d66cc53146ea01abe8612a6c5098 in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a9ab119 ]

[AIRFLOW-757] Set child_process_log_directory default more sensible

The default of child_process_log_directory was pointing to
/tmp/airflow/logs/scheduler. This could take people by surprise as
it is a non standard location and deviates from Airflow's other
log folders.


> Set sensible location for processor logs
> 
>
> Key: AIRFLOW-757
> URL: https://issues.apache.org/jira/browse/AIRFLOW-757
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> The current default location for processor logs is /tmp/airflow, this is 
> outside any of the standard locations and could surprise people. Its should 
> default to ~/airflow/logs/processor 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/5] incubator-airflow git commit: [AIRFLOW-757] Set child_process_log_directory default more sensible

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test df9464bba -> a46f114a5


[AIRFLOW-757] Set child_process_log_directory default more sensible

The default of child_process_log_directory was pointing to
/tmp/airflow/logs/scheduler. This could take people by surprise as
it is a non standard location and deviates from Airflow's other
log folders.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a9ab119b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a9ab119b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a9ab119b

Branch: refs/heads/v1-8-test
Commit: a9ab119b4081d66cc53146ea01abe8612a6c5098
Parents: 648bd4e
Author: Bolke de Bruin 
Authored: Sat Jan 14 13:58:38 2017 +0100
Committer: Bolke de Bruin 
Committed: Sat Jan 14 13:58:38 2017 +0100

--
 airflow/configuration.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a9ab119b/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 6778464..9b27328 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -349,7 +349,7 @@ dag_dir_list_interval = 300
 # How often should stats be printed to the logs
 print_stats_interval = 30
 
-child_process_log_directory = /tmp/airflow/scheduler/logs
+child_process_log_directory = {AIRFLOW_HOME}/logs/scheduler
 
 # Local task jobs periodically heartbeat to the DB. If the job has
 # not heartbeat in this many seconds, the scheduler will mark the

[jira] [Commented] (AIRFLOW-760) Update systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824604#comment-15824604
 ] 

ASF subversion and git services commented on AIRFLOW-760:
-

Commit 3ac2fba8885b64619ee9d622731708a4e78834e5 in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3ac2fba ]

Merge branch 'AIRFLOW-760'


> Update systemd scripts 
> ---
>
> Key: AIRFLOW-760
> URL: https://issues.apache.org/jira/browse/AIRFLOW-760
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> systemd scripts are not compatible with 1.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-760) Update systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824602#comment-15824602
 ] 

ASF subversion and git services commented on AIRFLOW-760:
-

Commit f3e18fbe022714f3a8a82040750db0b1ade9845a in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f3e18fb ]

[AIRFLOW-760] Update systemd config


> Update systemd scripts 
> ---
>
> Key: AIRFLOW-760
> URL: https://issues.apache.org/jira/browse/AIRFLOW-760
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> systemd scripts are not compatible with 1.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-757) Set sensible location for processor logs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824598#comment-15824598
 ] 

ASF subversion and git services commented on AIRFLOW-757:
-

Commit b0110bb4877a1f14c9e93fe29a0511914d1bb249 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b0110bb ]

Merge branch 'AIRFLOW-757'


> Set sensible location for processor logs
> 
>
> Key: AIRFLOW-757
> URL: https://issues.apache.org/jira/browse/AIRFLOW-757
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> The current default location for processor logs is /tmp/airflow, this is 
> outside any of the standard locations and could surprise people. Its should 
> default to ~/airflow/logs/processor 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-757) Set sensible location for processor logs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824596#comment-15824596
 ] 

ASF subversion and git services commented on AIRFLOW-757:
-

Commit a9ab119b4081d66cc53146ea01abe8612a6c5098 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a9ab119 ]

[AIRFLOW-757] Set child_process_log_directory default more sensible

The default of child_process_log_directory was pointing to
/tmp/airflow/logs/scheduler. This could take people by surprise as
it is a non standard location and deviates from Airflow's other
log folders.


> Set sensible location for processor logs
> 
>
> Key: AIRFLOW-757
> URL: https://issues.apache.org/jira/browse/AIRFLOW-757
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> The current default location for processor logs is /tmp/airflow, this is 
> outside any of the standard locations and could surprise people. Its should 
> default to ~/airflow/logs/processor 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/5] incubator-airflow git commit: Merge branch 'AIRFLOW-757'

[bolke]

Merge branch 'AIRFLOW-757'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b0110bb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b0110bb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b0110bb4

Branch: refs/heads/v1-8-test
Commit: b0110bb4877a1f14c9e93fe29a0511914d1bb249
Parents: 44798e0 a9ab119
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:23:27 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:23:27 2017 +0100

--
 airflow/configuration.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/4] incubator-airflow git commit: [AIRFLOW-757] Set child_process_log_directory default more sensible

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 44798e0d4 -> 3ac2fba88


[AIRFLOW-757] Set child_process_log_directory default more sensible

The default of child_process_log_directory was pointing to
/tmp/airflow/logs/scheduler. This could take people by surprise as
it is a non standard location and deviates from Airflow's other
log folders.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a9ab119b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a9ab119b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a9ab119b

Branch: refs/heads/master
Commit: a9ab119b4081d66cc53146ea01abe8612a6c5098
Parents: 648bd4e
Author: Bolke de Bruin 
Authored: Sat Jan 14 13:58:38 2017 +0100
Committer: Bolke de Bruin 
Committed: Sat Jan 14 13:58:38 2017 +0100

--
 airflow/configuration.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a9ab119b/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 6778464..9b27328 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -349,7 +349,7 @@ dag_dir_list_interval = 300
 # How often should stats be printed to the logs
 print_stats_interval = 30
 
-child_process_log_directory = /tmp/airflow/scheduler/logs
+child_process_log_directory = {AIRFLOW_HOME}/logs/scheduler
 
 # Local task jobs periodically heartbeat to the DB. If the job has
 # not heartbeat in this many seconds, the scheduler will mark the

[jira] [Commented] (AIRFLOW-760) Update systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824597#comment-15824597
 ] 

ASF subversion and git services commented on AIRFLOW-760:
-

Commit f3e18fbe022714f3a8a82040750db0b1ade9845a in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f3e18fb ]

[AIRFLOW-760] Update systemd config


> Update systemd scripts 
> ---
>
> Key: AIRFLOW-760
> URL: https://issues.apache.org/jira/browse/AIRFLOW-760
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> systemd scripts are not compatible with 1.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/5] incubator-airflow git commit: Merge branch 'AIRFLOW-760'

[bolke]

Merge branch 'AIRFLOW-760'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3ac2fba8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3ac2fba8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3ac2fba8

Branch: refs/heads/v1-8-test
Commit: 3ac2fba8885b64619ee9d622731708a4e78834e5
Parents: b0110bb f3e18fb
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:23:36 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:23:36 2017 +0100

--
 scripts/systemd/airflow-flower.service| 3 +--
 scripts/systemd/airflow-kerberos.service  | 3 +--
 scripts/systemd/airflow-scheduler.service | 1 -
 scripts/systemd/airflow-webserver.service | 7 ++-
 scripts/systemd/airflow-worker.service| 2 +-
 5 files changed, 5 insertions(+), 11 deletions(-)
--

[jira] [Commented] (AIRFLOW-760) Update systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824599#comment-15824599
 ] 

ASF subversion and git services commented on AIRFLOW-760:
-

Commit 3ac2fba8885b64619ee9d622731708a4e78834e5 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3ac2fba ]

Merge branch 'AIRFLOW-760'


> Update systemd scripts 
> ---
>
> Key: AIRFLOW-760
> URL: https://issues.apache.org/jira/browse/AIRFLOW-760
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> systemd scripts are not compatible with 1.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/5] incubator-airflow git commit: Merge branch 'master' into v1-8-test

[bolke]

Merge branch 'master' into v1-8-test


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a46f114a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a46f114a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a46f114a

Branch: refs/heads/v1-8-test
Commit: a46f114a5ea8f24dac1317204098153e8e930efa
Parents: df9464b 3ac2fba
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:24:20 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:24:20 2017 +0100

--
 airflow/configuration.py  | 2 +-
 scripts/systemd/airflow-flower.service| 3 +--
 scripts/systemd/airflow-kerberos.service  | 3 +--
 scripts/systemd/airflow-scheduler.service | 1 -
 scripts/systemd/airflow-webserver.service | 7 ++-
 scripts/systemd/airflow-worker.service| 2 +-
 6 files changed, 6 insertions(+), 12 deletions(-)
--

[2/5] incubator-airflow git commit: [AIRFLOW-760] Update systemd config

[bolke]

[AIRFLOW-760] Update systemd config


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f3e18fbe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f3e18fbe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f3e18fbe

Branch: refs/heads/v1-8-test
Commit: f3e18fbe022714f3a8a82040750db0b1ade9845a
Parents: 648bd4e
Author: Bolke de Bruin 
Authored: Sat Jan 14 21:32:27 2017 +0100
Committer: Bolke de Bruin 
Committed: Sat Jan 14 21:32:27 2017 +0100

--
 scripts/systemd/airflow-flower.service| 3 +--
 scripts/systemd/airflow-kerberos.service  | 3 +--
 scripts/systemd/airflow-scheduler.service | 1 -
 scripts/systemd/airflow-webserver.service | 7 ++-
 scripts/systemd/airflow-worker.service| 2 +-
 5 files changed, 5 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-flower.service
--
diff --git a/scripts/systemd/airflow-flower.service 
b/scripts/systemd/airflow-flower.service
index 4d524e8..de298c6 100644
--- a/scripts/systemd/airflow-flower.service
+++ b/scripts/systemd/airflow-flower.service
@@ -22,9 +22,8 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow flower
-KillMode=process
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-kerberos.service
--
diff --git a/scripts/systemd/airflow-kerberos.service 
b/scripts/systemd/airflow-kerberos.service
index 9c4ed6b..6113595 100644
--- a/scripts/systemd/airflow-kerberos.service
+++ b/scripts/systemd/airflow-kerberos.service
@@ -22,9 +22,8 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow kerberos
-KillMode=process
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-scheduler.service
--
diff --git a/scripts/systemd/airflow-scheduler.service 
b/scripts/systemd/airflow-scheduler.service
index 8c541cd..95229d6 100644
--- a/scripts/systemd/airflow-scheduler.service
+++ b/scripts/systemd/airflow-scheduler.service
@@ -22,7 +22,6 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS}
-KillMode=process
 Restart=always
 RestartSec=5s
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-webserver.service
--
diff --git a/scripts/systemd/airflow-webserver.service 
b/scripts/systemd/airflow-webserver.service
index 73445fe..e39083e 100644
--- a/scripts/systemd/airflow-webserver.service
+++ b/scripts/systemd/airflow-webserver.service
@@ -17,16 +17,13 @@ After=network.target postgresql.service mysql.service 
redis.service rabbitmq-ser
 Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
 
 [Service]
-PIDFile=/run/airflow/webserver.pid
 EnvironmentFile=/etc/sysconfig/airflow
 User=airflow
 Group=airflow
-Type=forking
+Type=simple
 ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid
-ExecReload=/bin/kill -s HUP $MAINPID
-ExecStop=/bin/kill -s TERM $MAINPID
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 PrivateTmp=true
 
 [Install]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-worker.service
--
diff --git a/scripts/systemd/airflow-worker.service 
b/scripts/systemd/airflow-worker.service
index 7531824..81ea353 100644
--- a/scripts/systemd/airflow-worker.service
+++ b/scripts/systemd/airflow-worker.service
@@ -23,7 +23,7 @@ Group=airflow
 Type=simple
 ExecStart=/bin/airflow worker
 Restart=on-failure
-RestartSec=42s
+RestartSec=10s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

[2/4] incubator-airflow git commit: [AIRFLOW-760] Update systemd config

[bolke]

[AIRFLOW-760] Update systemd config


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f3e18fbe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f3e18fbe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f3e18fbe

Branch: refs/heads/master
Commit: f3e18fbe022714f3a8a82040750db0b1ade9845a
Parents: 648bd4e
Author: Bolke de Bruin 
Authored: Sat Jan 14 21:32:27 2017 +0100
Committer: Bolke de Bruin 
Committed: Sat Jan 14 21:32:27 2017 +0100

--
 scripts/systemd/airflow-flower.service| 3 +--
 scripts/systemd/airflow-kerberos.service  | 3 +--
 scripts/systemd/airflow-scheduler.service | 1 -
 scripts/systemd/airflow-webserver.service | 7 ++-
 scripts/systemd/airflow-worker.service| 2 +-
 5 files changed, 5 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-flower.service
--
diff --git a/scripts/systemd/airflow-flower.service 
b/scripts/systemd/airflow-flower.service
index 4d524e8..de298c6 100644
--- a/scripts/systemd/airflow-flower.service
+++ b/scripts/systemd/airflow-flower.service
@@ -22,9 +22,8 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow flower
-KillMode=process
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-kerberos.service
--
diff --git a/scripts/systemd/airflow-kerberos.service 
b/scripts/systemd/airflow-kerberos.service
index 9c4ed6b..6113595 100644
--- a/scripts/systemd/airflow-kerberos.service
+++ b/scripts/systemd/airflow-kerberos.service
@@ -22,9 +22,8 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow kerberos
-KillMode=process
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-scheduler.service
--
diff --git a/scripts/systemd/airflow-scheduler.service 
b/scripts/systemd/airflow-scheduler.service
index 8c541cd..95229d6 100644
--- a/scripts/systemd/airflow-scheduler.service
+++ b/scripts/systemd/airflow-scheduler.service
@@ -22,7 +22,6 @@ User=airflow
 Group=airflow
 Type=simple
 ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS}
-KillMode=process
 Restart=always
 RestartSec=5s
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-webserver.service
--
diff --git a/scripts/systemd/airflow-webserver.service 
b/scripts/systemd/airflow-webserver.service
index 73445fe..e39083e 100644
--- a/scripts/systemd/airflow-webserver.service
+++ b/scripts/systemd/airflow-webserver.service
@@ -17,16 +17,13 @@ After=network.target postgresql.service mysql.service 
redis.service rabbitmq-ser
 Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
 
 [Service]
-PIDFile=/run/airflow/webserver.pid
 EnvironmentFile=/etc/sysconfig/airflow
 User=airflow
 Group=airflow
-Type=forking
+Type=simple
 ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid
-ExecReload=/bin/kill -s HUP $MAINPID
-ExecStop=/bin/kill -s TERM $MAINPID
 Restart=on-failure
-RestartSec=42s
+RestartSec=5s
 PrivateTmp=true
 
 [Install]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3e18fbe/scripts/systemd/airflow-worker.service
--
diff --git a/scripts/systemd/airflow-worker.service 
b/scripts/systemd/airflow-worker.service
index 7531824..81ea353 100644
--- a/scripts/systemd/airflow-worker.service
+++ b/scripts/systemd/airflow-worker.service
@@ -23,7 +23,7 @@ Group=airflow
 Type=simple
 ExecStart=/bin/airflow worker
 Restart=on-failure
-RestartSec=42s
+RestartSec=10s
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

[4/4] incubator-airflow git commit: Merge branch 'AIRFLOW-760'

[bolke]

Merge branch 'AIRFLOW-760'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3ac2fba8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3ac2fba8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3ac2fba8

Branch: refs/heads/master
Commit: 3ac2fba8885b64619ee9d622731708a4e78834e5
Parents: b0110bb f3e18fb
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:23:36 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:23:36 2017 +0100

--
 scripts/systemd/airflow-flower.service| 3 +--
 scripts/systemd/airflow-kerberos.service  | 3 +--
 scripts/systemd/airflow-scheduler.service | 1 -
 scripts/systemd/airflow-webserver.service | 7 ++-
 scripts/systemd/airflow-worker.service| 2 +-
 5 files changed, 5 insertions(+), 11 deletions(-)
--

[3/4] incubator-airflow git commit: Merge branch 'AIRFLOW-757'

[bolke]

Merge branch 'AIRFLOW-757'


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b0110bb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b0110bb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b0110bb4

Branch: refs/heads/master
Commit: b0110bb4877a1f14c9e93fe29a0511914d1bb249
Parents: 44798e0 a9ab119
Author: Bolke de Bruin 
Authored: Mon Jan 16 22:23:27 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 22:23:27 2017 +0100

--
 airflow/configuration.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (AIRFLOW-751) SFTP file transfer functionality

[Jayesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824443#comment-15824443
 ] 

Jayesh commented on AIRFLOW-751:


[~genomics-geek], I looked at your PR about sftp, while it has the required 
get/put functionality, I will have to refactor it based on the new ssh_hook 
from AIRFLOW-756

> SFTP file transfer functionality
> 
>
> Key: AIRFLOW-751
> URL: https://issues.apache.org/jira/browse/AIRFLOW-751
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Michael Gonzalez
>Assignee: Jayesh
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-756) Refactor ssh_hook and ssh_operator

[Jayesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824441#comment-15824441
 ] 

Jayesh commented on AIRFLOW-756:


I can't update this ticket, i also assigned this to me, but no luck.
anyway, summary is " Refactor ssh_hook and ssh_operator to use Paramiko instead 
of current subprocess"

> Refactor ssh_hook and ssh_operator
> --
>
> Key: AIRFLOW-756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-756
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks, operators
>Reporter: Michael Gonzalez
>Assignee: Jayesh
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (AIRFLOW-751) SFTP file transfer functionality

[Jayesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh reassigned AIRFLOW-751:
--

Assignee: Jayesh

> SFTP file transfer functionality
> 
>
> Key: AIRFLOW-751
> URL: https://issues.apache.org/jira/browse/AIRFLOW-751
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Michael Gonzalez
>Assignee: Jayesh
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (AIRFLOW-756) Refactor ssh_hook and ssh_operator

[Jayesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh reassigned AIRFLOW-756:
--

Assignee: Jayesh

> Refactor ssh_hook and ssh_operator
> --
>
> Key: AIRFLOW-756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-756
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks, operators
>Reporter: Michael Gonzalez
>Assignee: Jayesh
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-763) Vertica Check Operator

[Anmol Gautam (JIRA)]

Anmol Gautam created AIRFLOW-763:


 Summary: Vertica Check Operator
 Key: AIRFLOW-763
 URL: https://issues.apache.org/jira/browse/AIRFLOW-763
 Project: Apache Airflow
  Issue Type: New Feature
Reporter: Anmol Gautam
Assignee: Anmol Gautam
Priority: Minor


It will be good to have a Vertica Check Operator extending CheckOperator to 
handle checks on Vertica SQL queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-692) XCom page is visible to non-admin users

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824274#comment-15824274
 ] 

ASF subversion and git services commented on AIRFLOW-692:
-

Commit 648bd4e18d13feeecd6948f121b2772d9404afde in incubator-airflow's branch 
refs/heads/v1-8-test from [~msumit]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=648bd4e ]

[AIRFLOW-692] Open XCom page to super-admins only

Closes #1940 from msumit/AIRFLOW-692


> XCom page is visible to non-admin users
> ---
>
> Key: AIRFLOW-692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>  Labels: UI
>
> XCom page should not be visible to normal users. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-683) provide Jira Hook, Operator & Sensor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824278#comment-15824278
 ] 

ASF subversion and git services commented on AIRFLOW-683:
-

Commit 44798e0d4d36e6a793d19b4986f2ddb6814ec208 in incubator-airflow's branch 
refs/heads/v1-8-test from [~jhsenjaliya]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=44798e0 ]

[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


> provide Jira Hook, Operator & Sensor
> 
>
> Key: AIRFLOW-683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-683
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Reporter: Jayesh
>Assignee: Jayesh
>
> Task to add Jira Hook, Operator & Sensor so that user can interact with Jira 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-762) Google DataProc cluster delete operator

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824276#comment-15824276
 ] 

ASF subversion and git services commented on AIRFLOW-762:
-

Commit a8b2f7f26fb8ca16b8ecdfb40ec2e85210fdf75e in incubator-airflow's branch 
refs/heads/v1-8-test from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8b2f7f ]

[AIRFLOW-762] Add Google DataProc delete operator

Pair the recently added Create operator with a
Delete operator for
Google Cloud DataProc clusters.

Closes #1997 from alexvanboxel/pr/dataproc


> Google DataProc cluster delete operator
> ---
>
> Key: AIRFLOW-762
> URL: https://issues.apache.org/jira/browse/AIRFLOW-762
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>
> Add a delete cluster operator for Google Cloud DataProc



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-759) The dependency checker should verify if the interval is the first interval after start_date

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824275#comment-15824275
 ] 

ASF subversion and git services commented on AIRFLOW-759:
-

Commit 89f0ca4abfa38b66d2e26788e353bfdd17772c52 in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=89f0ca4 ]

[AIRFLOW-759] Use previous dag_run to verify depend_on_past

The start_date and the schedule interval can be misaligned. This
is automatically corrected in the scheduler. The dependency checker
however did not do this.


> The dependency checker should verify if the interval is the first interval 
> after start_date 
> 
>
> Key: AIRFLOW-759
> URL: https://issues.apache.org/jira/browse/AIRFLOW-759
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> Since 1.8 we support auto aligned start_date and interval. This is not 
> reflected in the dependency checker and as such depend_on_past tasks will 
> fail if the start_date and the execution_date are not aligned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-683) provide Jira Hook, Operator & Sensor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824277#comment-15824277
 ] 

ASF subversion and git services commented on AIRFLOW-683:
-

Commit 44798e0d4d36e6a793d19b4986f2ddb6814ec208 in incubator-airflow's branch 
refs/heads/v1-8-test from [~jhsenjaliya]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=44798e0 ]

[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


> provide Jira Hook, Operator & Sensor
> 
>
> Key: AIRFLOW-683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-683
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Reporter: Jayesh
>Assignee: Jayesh
>
> Task to add Jira Hook, Operator & Sensor so that user can interact with Jira 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-692) XCom page is visible to non-admin users

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824273#comment-15824273
 ] 

ASF subversion and git services commented on AIRFLOW-692:
-

Commit 648bd4e18d13feeecd6948f121b2772d9404afde in incubator-airflow's branch 
refs/heads/v1-8-test from [~msumit]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=648bd4e ]

[AIRFLOW-692] Open XCom page to super-admins only

Closes #1940 from msumit/AIRFLOW-692


> XCom page is visible to non-admin users
> ---
>
> Key: AIRFLOW-692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>  Labels: UI
>
> XCom page should not be visible to normal users. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[6/9] incubator-airflow git commit: Merge remote-tracking branch 'apache/master' into v1-8-test

[bolke]

Merge remote-tracking branch 'apache/master' into v1-8-test


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a41de048
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a41de048
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a41de048

Branch: refs/heads/v1-8-test
Commit: a41de048156203f23ab5e14dbf3ab06770998895
Parents: bd08dcc a8b2f7f
Author: Bolke de Bruin 
Authored: Mon Jan 16 17:40:18 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:40:18 2017 +0100

--
 airflow/contrib/operators/dataproc_operator.py | 72 +
 1 file changed, 72 insertions(+)
--

[8/9] incubator-airflow git commit: [AIRFLOW-683] Add jira hook, operator and sensor

[bolke]

[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/44798e0d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/44798e0d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/44798e0d

Branch: refs/heads/v1-8-test
Commit: 44798e0d4d36e6a793d19b4986f2ddb6814ec208
Parents: a8b2f7f
Author: Jay 
Authored: Mon Jan 16 17:46:11 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:46:21 2017 +0100

--
 airflow/contrib/hooks/jira_hook.py|  82 
 airflow/contrib/operators/jira_operator.py|  89 +
 airflow/contrib/sensors/jira_sensor.py| 146 +
 airflow/models.py |   4 +
 scripts/ci/requirements.txt   |   1 +
 setup.py  |   2 +
 tests/contrib/hooks/test_jira_hook.py |  51 +++
 tests/contrib/operators/jira_operator_test.py | 101 ++
 tests/contrib/sensors/jira_sensor_test.py |  85 
 9 files changed, 561 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/44798e0d/airflow/contrib/hooks/jira_hook.py
--
diff --git a/airflow/contrib/hooks/jira_hook.py 
b/airflow/contrib/hooks/jira_hook.py
new file mode 100644
index 000..148101b
--- /dev/null
+++ b/airflow/contrib/hooks/jira_hook.py
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from jira import JIRA
+from jira.exceptions import JIRAError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class JiraHook(BaseHook):
+"""
+Jira interaction hook, a Wrapper around JIRA Python SDK.
+
+:param jira_conn_id: reference to a pre-defined Jira Connection
+:type jira_conn_id: string
+"""
+
+def __init__(self,
+ jira_conn_id='jira_default'):
+super(JiraHook, self).__init__(jira_conn_id)
+self.jira_conn_id = jira_conn_id
+self.client = None
+self.get_conn()
+
+def get_conn(self):
+if not self.client:
+logging.debug('creating jira client for conn_id: 
{0}'.format(self.jira_conn_id))
+
+get_server_info = True
+validate = True
+extra_options = {}
+conn = None
+
+if self.jira_conn_id is not None:
+conn = self.get_connection(self.jira_conn_id)
+if conn.extra is not None:
+extra_options = conn.extra_dejson
+# only required attributes are taken for now,
+# more can be added ex: async, logging, max_retries
+
+# verify
+if 'verify' in extra_options \
+and extra_options['verify'].lower() == 'false':
+extra_options['verify'] = False
+
+# validate
+if 'validate' in extra_options \
+and extra_options['validate'].lower() == 'false':
+validate = False
+
+if 'get_server_info' in extra_options \
+and extra_options['get_server_info'].lower() == 
'false':
+get_server_info = False
+
+try:
+self.client = JIRA(conn.host,
+   options=extra_options,
+   basic_auth=(conn.login, conn.password),
+   get_server_info=get_server_info,
+   validate=validate)
+except JIRAError as jira_error:
+raise AirflowException('Failed to create jira client, jira 
error: %s'
+   % str(jira_error))
+except Exception as e:
+raise AirflowException('Failed to create jira client, 
error: %s'
+   % str(e))
+
+return self.client

[5/9] incubator-airflow git commit: Merge branch 'master' into v1-8-test

[bolke]

Merge branch 'master' into v1-8-test


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/bd08dcc5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/bd08dcc5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/bd08dcc5

Branch: refs/heads/v1-8-test
Commit: bd08dcc54d7b8791eef8956baf58e4163d9666ee
Parents: 5782ad4 89f0ca4
Author: Bolke de Bruin 
Authored: Mon Jan 16 17:40:07 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:40:07 2017 +0100

--
 airflow/contrib/sensors/hdfs_sensors.py |  1 +
 airflow/models.py   | 46 
 airflow/ti_deps/deps/prev_dagrun_dep.py |  4 +--
 airflow/www/views.py|  2 +-
 tests/core.py   |  8 ++---
 5 files changed, 34 insertions(+), 27 deletions(-)
--

[4/9] incubator-airflow git commit: [AIRFLOW-762] Add Google DataProc delete operator

[bolke]

[AIRFLOW-762] Add Google DataProc delete operator

Pair the recently added Create operator with a
Delete operator for
Google Cloud DataProc clusters.

Closes #1997 from alexvanboxel/pr/dataproc


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a8b2f7f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a8b2f7f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a8b2f7f2

Branch: refs/heads/v1-8-test
Commit: a8b2f7f26fb8ca16b8ecdfb40ec2e85210fdf75e
Parents: 89f0ca4
Author: Alex Van Boxel 
Authored: Mon Jan 16 17:38:28 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:38:28 2017 +0100

--
 airflow/contrib/operators/dataproc_operator.py | 72 +
 1 file changed, 72 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a8b2f7f2/airflow/contrib/operators/dataproc_operator.py
--
diff --git a/airflow/contrib/operators/dataproc_operator.py 
b/airflow/contrib/operators/dataproc_operator.py
index 9cf2bbe..24fa2e4 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -270,6 +270,78 @@ class DataprocClusterCreateOperator(BaseOperator):
 self._wait_for_done(service)
 
 
+class DataprocClusterDeleteOperator(BaseOperator):
+"""
+Delete a cluster on Google Cloud Dataproc. The operator will wait until the
+cluster is destroyed.
+"""
+
+template_fields = ['cluster_name']
+
+@apply_defaults
+def __init__(self,
+ cluster_name,
+ project_id,
+ region='global',
+ google_cloud_conn_id='google_cloud_default',
+ delegate_to=None,
+ *args,
+ **kwargs):
+"""
+Delete a cluster on Google Cloud Dataproc.
+
+:param cluster_name: The name of the cluster to create
+:type cluster_name: string
+:param project_id: The ID of the google cloud project in which
+the cluster runs
+:type project_id: string
+:param region: leave as 'global', might become relevant in the future
+:type region: string
+:param google_cloud_conn_id: The connection id to use when connecting 
to dataproc
+:type google_cloud_conn_id: string
+:param delegate_to: The account to impersonate, if any.
+For this to work, the service account making the request must have 
domain-wide
+delegation enabled.
+:type delegate_to: string
+"""
+super(DataprocClusterDeleteOperator, self).__init__(*args, **kwargs)
+self.google_cloud_conn_id = google_cloud_conn_id
+self.delegate_to = delegate_to
+self.cluster_name = cluster_name
+self.project_id = project_id
+self.region = region
+
+def _wait_for_done(self, service, operation_name):
+time.sleep(15)
+while True:
+response = service.projects().regions().operations().get(
+name=operation_name
+).execute()
+
+if 'done' in response and response['done']:
+if 'error' in response:
+raise Exception(str(response['error']))
+else:
+return
+time.sleep(15)
+
+def execute(self, context):
+hook = DataProcHook(
+gcp_conn_id=self.google_cloud_conn_id,
+delegate_to=self.delegate_to
+)
+service = hook.get_conn()
+
+response = service.projects().regions().clusters().delete(
+projectId=self.project_id,
+region=self.region,
+clusterName=self.cluster_name
+).execute()
+operation_name = response['name']
+logging.info("Cluster delete operation name: 
{}".format(operation_name))
+self._wait_for_done(service, operation_name)
+
+
 class DataProcPigOperator(BaseOperator):
 """
 Start a Pig query Job on a Cloud DataProc cluster. The parameters of the 
operation

[jira] [Closed] (AIRFLOW-683) provide Jira Hook, Operator & Sensor

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin closed AIRFLOW-683.
--
Resolution: Fixed

> provide Jira Hook, Operator & Sensor
> 
>
> Key: AIRFLOW-683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-683
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Reporter: Jayesh
>Assignee: Jayesh
>
> Task to add Jira Hook, Operator & Sensor so that user can interact with Jira 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/9] incubator-airflow git commit: [AIRFLOW-692] Open XCom page to super-admins only

[bolke]

[AIRFLOW-692] Open XCom page to super-admins only

Closes #1940 from msumit/AIRFLOW-692


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/648bd4e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/648bd4e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/648bd4e1

Branch: refs/heads/v1-8-test
Commit: 648bd4e18d13feeecd6948f121b2772d9404afde
Parents: 8fd65f8
Author: Sumit Maheshwari 
Authored: Fri Jan 13 21:17:04 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Jan 13 21:17:04 2017 +0100

--
 airflow/www/views.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/648bd4e1/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 5de6855..800275a 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2249,7 +2249,7 @@ class VariableView(wwwutils.DataProfilingMixin, 
AirflowModelView):
 form.val.data = '*' * 8
 
 
-class XComView(wwwutils.LoginMixin, AirflowModelView):
+class XComView(wwwutils.SuperUserMixin, AirflowModelView):
 verbose_name = "XCom"
 verbose_name_plural = "XComs"
 page_size = 20

[9/9] incubator-airflow git commit: Merge branch 'master' into v1-8-test

[bolke]

Merge branch 'master' into v1-8-test


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/df9464bb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/df9464bb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/df9464bb

Branch: refs/heads/v1-8-test
Commit: df9464bbad7c275125240b8fa4de577edd4143f4
Parents: 36b16a5 44798e0
Author: Bolke de Bruin 
Authored: Mon Jan 16 17:48:13 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:48:13 2017 +0100

--
 airflow/contrib/hooks/jira_hook.py|  82 
 airflow/contrib/operators/jira_operator.py|  89 +
 airflow/contrib/sensors/jira_sensor.py| 146 +
 airflow/models.py |   4 +
 scripts/ci/requirements.txt   |   1 +
 setup.py  |   2 +
 tests/contrib/hooks/test_jira_hook.py |  51 +++
 tests/contrib/operators/jira_operator_test.py | 101 ++
 tests/contrib/sensors/jira_sensor_test.py |  85 
 9 files changed, 561 insertions(+)
--

[7/9] incubator-airflow git commit: Bump version to 1.8.0b1

[bolke]

Bump version to 1.8.0b1


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/36b16a56
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/36b16a56
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/36b16a56

Branch: refs/heads/v1-8-test
Commit: 36b16a5617f95b5f34e9735ec7e08816de21edb0
Parents: a41de04
Author: Bolke de Bruin 
Authored: Mon Jan 16 17:45:07 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:45:07 2017 +0100

--
 airflow/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/36b16a56/airflow/version.py
--
diff --git a/airflow/version.py b/airflow/version.py
index d444304..3d3b0c3 100644
--- a/airflow/version.py
+++ b/airflow/version.py
@@ -13,4 +13,4 @@
 # limitations under the License.
 #
 
-version = '1.8.0a5'
+version = '1.8.0b1'

[3/9] incubator-airflow git commit: [AIRFLOW-759] Use previous dag_run to verify depend_on_past

[bolke]

[AIRFLOW-759] Use previous dag_run to verify depend_on_past

The start_date and the schedule interval can be misaligned. This
is automatically corrected in the scheduler. The dependency checker
however did not do this.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/89f0ca4a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/89f0ca4a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/89f0ca4a

Branch: refs/heads/v1-8-test
Commit: 89f0ca4abfa38b66d2e26788e353bfdd17772c52
Parents: 648bd4e
Author: Bolke de Bruin 
Authored: Sat Jan 14 14:31:09 2017 +0100
Committer: Bolke de Bruin 
Committed: Sat Jan 14 21:10:56 2017 +0100

--
 airflow/models.py   | 46 
 airflow/ti_deps/deps/prev_dagrun_dep.py |  4 +--
 2 files changed, 28 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/89f0ca4a/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index acb6667..d878457 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1027,19 +1027,22 @@ class TaskInstance(Base):
 dag = self.task.dag
 if dag:
 dr = self.get_dagrun(session=session)
+
+# LEGACY: most likely running from unit tests
 if not dr:
 # Means that this TI is NOT being run from a DR, but from a 
catchup
 previous_scheduled_date = 
dag.previous_schedule(self.execution_date)
 if not previous_scheduled_date:
 return None
-else:
-return TaskInstance(task=self.task, 
execution_date=previous_scheduled_date)
 
-if dag.catchup:
-last_dagrun = 
dr.get_previous_scheduled_dagrun(session=session) if dr else None
+return TaskInstance(task=self.task,
+execution_date=previous_scheduled_date)
 
+dr.dag = dag
+if dag.catchup:
+last_dagrun = dr.get_previous_scheduled_dagrun(session=session)
 else:
-last_dagrun = dr.get_previous_dagrun(session=session) if dr 
else None
+last_dagrun = dr.get_previous_dagrun(session=session)
 
 if last_dagrun:
 return last_dagrun.get_task_instance(self.task_id, 
session=session)
@@ -1066,16 +1069,21 @@ class TaskInstance(Base):
 :type verbose: boolean
 """
 dep_context = dep_context or DepContext()
+failed = False
 for dep_status in self.get_failed_dep_statuses(
 dep_context=dep_context,
 session=session):
+failed = True
 if verbose:
-logging.warning(
-"Dependencies not met for %s, dependency '%s' FAILED: %s",
-self, dep_status.dep_name, dep_status.reason)
+logging.info("Dependencies not met for {}, dependency '{}' 
FAILED: {}"
+ .format(self, dep_status.dep_name, 
dep_status.reason))
+
+if failed:
 return False
+
 if verbose:
-logging.info("Dependencies all met for %s", self)
+logging.info("Dependencies all met for {}".format(self))
+
 return True
 
 @provide_session
@@ -1089,12 +1097,14 @@ class TaskInstance(Base):
 self,
 session,
 dep_context):
-if dep_status.passed:
-logging.debug("%s dependency '%s' PASSED: %s",
-  self,
-  dep_status.dep_name,
-  dep_status.reason)
-else:
+
+logging.debug("{} dependency '{}' PASSED: {}, {}"
+  .format(self,
+  dep_status.dep_name,
+  dep_status.passed,
+  dep_status.reason))
+
+if not dep_status.passed:
 yield dep_status
 
 def __repr__(self):
@@ -3882,13 +3892,11 @@ class DagRun(Base):
 @provide_session
 def get_previous_scheduled_dagrun(self, session=None):
 """The previous, SCHEDULED DagRun, if there is one"""
-
-if not self.dag:
-return None
+dag = self.get_dag()
 
 return session.query(DagRun).filter(
 DagRun.dag_id == self.dag_id,
-DagRun.execution_date == 
self.dag.previous_schedule(self.execution_date)
+DagRun.execution_date == 

[1/9] incubator-airflow git commit: [AIRFLOW-737] Fix HDFS Sensor directory.

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test 5782ad4b6 -> df9464bba


[AIRFLOW-737] Fix HDFS Sensor directory.

Due to a bad ordering in the fake snakebite
client, one test was wrongly True.

Closes #1989 from
vfoucault/fixbug/hdfssensor_folder


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8fd65f8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8fd65f8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8fd65f8a

Branch: refs/heads/v1-8-test
Commit: 8fd65f8ac4a305da01324c5120fbfab8b51f5d46
Parents: 4a5e17b
Author: Vianney Foucault 
Authored: Fri Jan 13 21:15:14 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Jan 13 21:15:14 2017 +0100

--
 airflow/contrib/sensors/hdfs_sensors.py | 1 +
 tests/core.py   | 8 
 2 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8fd65f8a/airflow/contrib/sensors/hdfs_sensors.py
--
diff --git a/airflow/contrib/sensors/hdfs_sensors.py 
b/airflow/contrib/sensors/hdfs_sensors.py
index a12e295..4e9bb9b 100644
--- a/airflow/contrib/sensors/hdfs_sensors.py
+++ b/airflow/contrib/sensors/hdfs_sensors.py
@@ -62,6 +62,7 @@ class HdfsSensorFolder(HdfsSensor):
 return len(result) == 1 and result[0]['path'] == self.filepath
 else:
 logging.info('Poking for filepath {self.filepath} to a non empty 
directory'.format(**locals()))
+result.pop(0)
 return bool(result) and result[0]['file_type'] == 'f'
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8fd65f8a/tests/core.py
--
diff --git a/tests/core.py b/tests/core.py
index 6fb735f..0f7e41d 100644
--- a/tests/core.py
+++ b/tests/core.py
@@ -1840,10 +1840,6 @@ class FakeSnakeBiteClient(object):
 {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 
'access_time': 0, 'block_replication': 0,
  'modification_time': 1481132141540, 'length': 0, 'blocksize': 
0, 'owner': u'hdfs',
  'path': '/datadirectory/empty_directory'}]
-elif path[0] == '/datadirectory/not_empty_directory':
-return [{'group': u'supergroup', 'permission': 420, 'file_type': 
'f', 'access_time': 1481122343796,
- 'block_replication': 3, 'modification_time': 
1481122343862, 'length': 0, 'blocksize': 134217728,
- 'owner': u'hdfs', 'path': 
'/datadirectory/not_empty_directory/test_file'}]
 elif path[0] == '/datadirectory/not_empty_directory' and 
include_toplevel:
 return [
 {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 
'access_time': 0, 'block_replication': 0,
@@ -1852,6 +1848,10 @@ class FakeSnakeBiteClient(object):
 {'group': u'supergroup', 'permission': 420, 'file_type': 'f', 
'access_time': 1481122343796,
  'block_replication': 3, 'modification_time': 1481122343862, 
'length': 0, 'blocksize': 134217728,
  'owner': u'hdfs', 'path': 
'/datadirectory/not_empty_directory/test_file'}]
+elif path[0] == '/datadirectory/not_empty_directory':
+return [{'group': u'supergroup', 'permission': 420, 'file_type': 
'f', 'access_time': 1481122343796,
+ 'block_replication': 3, 'modification_time': 
1481122343862, 'length': 0, 'blocksize': 134217728,
+ 'owner': u'hdfs', 'path': 
'/datadirectory/not_empty_directory/test_file'}]
 elif path[0] == '/datadirectory/not_existing_file_or_directory':
 raise FakeSnakeBiteClientException
 elif path[0] == '/datadirectory/regex_dir':

[jira] [Commented] (AIRFLOW-683) provide Jira Hook, Operator & Sensor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824268#comment-15824268
 ] 

ASF subversion and git services commented on AIRFLOW-683:
-

Commit 44798e0d4d36e6a793d19b4986f2ddb6814ec208 in incubator-airflow's branch 
refs/heads/master from [~jhsenjaliya]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=44798e0 ]

[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


> provide Jira Hook, Operator & Sensor
> 
>
> Key: AIRFLOW-683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-683
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Reporter: Jayesh
>Assignee: Jayesh
>
> Task to add Jira Hook, Operator & Sensor so that user can interact with Jira 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-683) provide Jira Hook, Operator & Sensor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824269#comment-15824269
 ] 

ASF subversion and git services commented on AIRFLOW-683:
-

Commit 44798e0d4d36e6a793d19b4986f2ddb6814ec208 in incubator-airflow's branch 
refs/heads/master from [~jhsenjaliya]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=44798e0 ]

[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


> provide Jira Hook, Operator & Sensor
> 
>
> Key: AIRFLOW-683
> URL: https://issues.apache.org/jira/browse/AIRFLOW-683
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Reporter: Jayesh
>Assignee: Jayesh
>
> Task to add Jira Hook, Operator & Sensor so that user can interact with Jira 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-airflow git commit: [AIRFLOW-683] Add jira hook, operator and sensor

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master a8b2f7f26 -> 44798e0d4


[AIRFLOW-683] Add jira hook, operator and sensor

Closes #1950 from jhsenjaliya/AIRFLOW-683


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/44798e0d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/44798e0d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/44798e0d

Branch: refs/heads/master
Commit: 44798e0d4d36e6a793d19b4986f2ddb6814ec208
Parents: a8b2f7f
Author: Jay 
Authored: Mon Jan 16 17:46:11 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:46:21 2017 +0100

--
 airflow/contrib/hooks/jira_hook.py|  82 
 airflow/contrib/operators/jira_operator.py|  89 +
 airflow/contrib/sensors/jira_sensor.py| 146 +
 airflow/models.py |   4 +
 scripts/ci/requirements.txt   |   1 +
 setup.py  |   2 +
 tests/contrib/hooks/test_jira_hook.py |  51 +++
 tests/contrib/operators/jira_operator_test.py | 101 ++
 tests/contrib/sensors/jira_sensor_test.py |  85 
 9 files changed, 561 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/44798e0d/airflow/contrib/hooks/jira_hook.py
--
diff --git a/airflow/contrib/hooks/jira_hook.py 
b/airflow/contrib/hooks/jira_hook.py
new file mode 100644
index 000..148101b
--- /dev/null
+++ b/airflow/contrib/hooks/jira_hook.py
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from jira import JIRA
+from jira.exceptions import JIRAError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class JiraHook(BaseHook):
+"""
+Jira interaction hook, a Wrapper around JIRA Python SDK.
+
+:param jira_conn_id: reference to a pre-defined Jira Connection
+:type jira_conn_id: string
+"""
+
+def __init__(self,
+ jira_conn_id='jira_default'):
+super(JiraHook, self).__init__(jira_conn_id)
+self.jira_conn_id = jira_conn_id
+self.client = None
+self.get_conn()
+
+def get_conn(self):
+if not self.client:
+logging.debug('creating jira client for conn_id: 
{0}'.format(self.jira_conn_id))
+
+get_server_info = True
+validate = True
+extra_options = {}
+conn = None
+
+if self.jira_conn_id is not None:
+conn = self.get_connection(self.jira_conn_id)
+if conn.extra is not None:
+extra_options = conn.extra_dejson
+# only required attributes are taken for now,
+# more can be added ex: async, logging, max_retries
+
+# verify
+if 'verify' in extra_options \
+and extra_options['verify'].lower() == 'false':
+extra_options['verify'] = False
+
+# validate
+if 'validate' in extra_options \
+and extra_options['validate'].lower() == 'false':
+validate = False
+
+if 'get_server_info' in extra_options \
+and extra_options['get_server_info'].lower() == 
'false':
+get_server_info = False
+
+try:
+self.client = JIRA(conn.host,
+   options=extra_options,
+   basic_auth=(conn.login, conn.password),
+   get_server_info=get_server_info,
+   validate=validate)
+except JIRAError as jira_error:
+raise AirflowException('Failed to create jira client, jira 
error: %s'
+   % str(jira_error))
+except Exception as e:
+raise AirflowException('Failed to create jira client, 
error: 

[jira] [Resolved] (AIRFLOW-762) Google DataProc cluster delete operator

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-762.

Resolution: Fixed

Issue resolved by pull request #1997
[https://github.com/apache/incubator-airflow/pull/1997]

> Google DataProc cluster delete operator
> ---
>
> Key: AIRFLOW-762
> URL: https://issues.apache.org/jira/browse/AIRFLOW-762
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>
> Add a delete cluster operator for Google Cloud DataProc



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-762) Google DataProc cluster delete operator

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824244#comment-15824244
 ] 

ASF subversion and git services commented on AIRFLOW-762:
-

Commit a8b2f7f26fb8ca16b8ecdfb40ec2e85210fdf75e in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8b2f7f ]

[AIRFLOW-762] Add Google DataProc delete operator

Pair the recently added Create operator with a
Delete operator for
Google Cloud DataProc clusters.

Closes #1997 from alexvanboxel/pr/dataproc


> Google DataProc cluster delete operator
> ---
>
> Key: AIRFLOW-762
> URL: https://issues.apache.org/jira/browse/AIRFLOW-762
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>
> Add a delete cluster operator for Google Cloud DataProc



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-airflow git commit: [AIRFLOW-762] Add Google DataProc delete operator

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 89f0ca4ab -> a8b2f7f26


[AIRFLOW-762] Add Google DataProc delete operator

Pair the recently added Create operator with a
Delete operator for
Google Cloud DataProc clusters.

Closes #1997 from alexvanboxel/pr/dataproc


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a8b2f7f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a8b2f7f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a8b2f7f2

Branch: refs/heads/master
Commit: a8b2f7f26fb8ca16b8ecdfb40ec2e85210fdf75e
Parents: 89f0ca4
Author: Alex Van Boxel 
Authored: Mon Jan 16 17:38:28 2017 +0100
Committer: Bolke de Bruin 
Committed: Mon Jan 16 17:38:28 2017 +0100

--
 airflow/contrib/operators/dataproc_operator.py | 72 +
 1 file changed, 72 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a8b2f7f2/airflow/contrib/operators/dataproc_operator.py
--
diff --git a/airflow/contrib/operators/dataproc_operator.py 
b/airflow/contrib/operators/dataproc_operator.py
index 9cf2bbe..24fa2e4 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -270,6 +270,78 @@ class DataprocClusterCreateOperator(BaseOperator):
 self._wait_for_done(service)
 
 
+class DataprocClusterDeleteOperator(BaseOperator):
+"""
+Delete a cluster on Google Cloud Dataproc. The operator will wait until the
+cluster is destroyed.
+"""
+
+template_fields = ['cluster_name']
+
+@apply_defaults
+def __init__(self,
+ cluster_name,
+ project_id,
+ region='global',
+ google_cloud_conn_id='google_cloud_default',
+ delegate_to=None,
+ *args,
+ **kwargs):
+"""
+Delete a cluster on Google Cloud Dataproc.
+
+:param cluster_name: The name of the cluster to create
+:type cluster_name: string
+:param project_id: The ID of the google cloud project in which
+the cluster runs
+:type project_id: string
+:param region: leave as 'global', might become relevant in the future
+:type region: string
+:param google_cloud_conn_id: The connection id to use when connecting 
to dataproc
+:type google_cloud_conn_id: string
+:param delegate_to: The account to impersonate, if any.
+For this to work, the service account making the request must have 
domain-wide
+delegation enabled.
+:type delegate_to: string
+"""
+super(DataprocClusterDeleteOperator, self).__init__(*args, **kwargs)
+self.google_cloud_conn_id = google_cloud_conn_id
+self.delegate_to = delegate_to
+self.cluster_name = cluster_name
+self.project_id = project_id
+self.region = region
+
+def _wait_for_done(self, service, operation_name):
+time.sleep(15)
+while True:
+response = service.projects().regions().operations().get(
+name=operation_name
+).execute()
+
+if 'done' in response and response['done']:
+if 'error' in response:
+raise Exception(str(response['error']))
+else:
+return
+time.sleep(15)
+
+def execute(self, context):
+hook = DataProcHook(
+gcp_conn_id=self.google_cloud_conn_id,
+delegate_to=self.delegate_to
+)
+service = hook.get_conn()
+
+response = service.projects().regions().clusters().delete(
+projectId=self.project_id,
+region=self.region,
+clusterName=self.cluster_name
+).execute()
+operation_name = response['name']
+logging.info("Cluster delete operation name: 
{}".format(operation_name))
+self._wait_for_done(service, operation_name)
+
+
 class DataProcPigOperator(BaseOperator):
 """
 Start a Pig query Job on a Cloud DataProc cluster. The parameters of the 
operation