[jira] [Assigned] (AIRFLOW-2343) pip gcp_api dependency conflicts

2018-04-27 Thread Kaxil Naik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik reassigned AIRFLOW-2343:
---

Assignee: Kaxil Naik

> pip gcp_api dependency conflicts
> 
>
> Key: AIRFLOW-2343
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2343
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Affects Versions: Airflow 1.9.0
> Environment: Ubuntu Server 17.10
> Debian 9
>Reporter: Simon Hellbe
>Assignee: Kaxil Naik
>Priority: Major
>
> In a clean virtualenv with python 2.7 on Ubuntu:
> {code:java}
> pip install apache-airflow[gcp_api]{code}
> Fails with downstream dependency conflicts for the installed versions of 
> google-cloud-bigquery and google-cloud-pubsub. Have tried installing various 
> versions of the packages but can't find a compatible combination.
> {code:java}
> google-cloud-pubsub 0.26.0 has requirement 
> google-cloud-core<0.26dev,>=0.25.0, but you'll have google-cloud-core 0.28.1 
> which is incompatible.
> apache-beam 2.4.0 has requirement dill==0.2.6, but you'll have dill 0.2.7.1 
> which is incompatible.apache-beam 2.4.0 has requirement httplib2<0.10,>=0.8, 
> but you'll have httplib2 0.11.3 which is incompatible.
> googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
> httplib2 0.11.3 which is incompatible.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-2266) GCP integrations (gcp_api extras) cannot be installed from Python 3

2018-04-27 Thread Kaxil Naik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik reassigned AIRFLOW-2266:
---

Assignee: Kaxil Naik

> GCP integrations (gcp_api extras) cannot be installed from Python 3
> ---
>
> Key: AIRFLOW-2266
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2266
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
> Environment: OS: macOS
> $ sw_vers
> ProductName:  Mac OS X
> ProductVersion:   10.13.3
> BuildVersion: 17D102
> Python 3.6.4
>Reporter: Tim Swast`
>Assignee: Kaxil Naik
>Priority: Major
>
> As seen in https://issues.apache.org/jira/browse/AIRFLOW-2187, the 
> google-cloud-dataflow package only works from Python 2. This is what happens 
> when I try to install the gcp_api extras from a Python 3 environment:
>  
> |pip install apache-airflow[gcp_api]|
> |...
>  Collecting cryptography>=2.1.4 (from PyOpenSSL->apache-airflow[gcp_api])
>   Downloading cryptography-2.2.1-cp34-abi3-macosx_10_6_intel.whl (1.5MB)
>     100% \|\| 1.5MB 943kB/s
>  Collecting apache-beam[gcp]==2.3.0 (from 
> google-cloud-dataflow->apache-airflow[gcp_api])
>   Could not find a version that satisfies the requirement 
> apache-beam[gcp]==2.3.0 (from google-cloud-dataflow->apache-airflow[gcp_api]) 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
>  No matching distribution found for apache-beam[gcp]==2.3.0 (from 
> google-cloud-dataflow->apache-airflow[gcp_api])|
>  
> The other GCP APIs should be accessible from Python 3. My suggestion is to 
> split the gcp_extras into separate extras (or at least a separate extra for 
> dataflow), similar to how there is a separate extra for s3. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1970) Database cannot be initialized if an invalid fernet key is provided

2018-04-27 Thread Ramki Subramanian (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457240#comment-16457240
 ] 

Ramki Subramanian commented on AIRFLOW-1970:


I got a similar error. Make sure

 
 # The generated fernet key ( from the code snippet) is copied/ pasted properly 
in the config file
 # Restart airflow webserver + scheduler. 

It should solve the issue. 

> Database cannot be initialized if an invalid fernet key is provided
> ---
>
> Key: AIRFLOW-1970
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1970
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.9.0
> Environment: Python 2.7.12
> PostgreSQL 9.6.3
>Reporter: Michael Backes
>Priority: Major
>
> If I use an invalid fernet key in my config file, I'm not able to run 
> "airflow initdb" successfully.
> For example if I have the following in my config:
> {panel:title=airflow.cfg}
> ...
> \\# Secret key to save connection passwords in the db
> fernet_key = xxx
> ...
> {panel}
> I will get the following error when running "airflow initdb":
> {panel:title=log}
> [2018-01-05 16:43:00,525] {__init__.py:45} INFO - Using executor LocalExecutor
> DB: postgresql+psycopg2://airflow_user:secret_pw@some.address:5432/airflow
> [2018-01-05 16:43:00,624] {db.py:312} INFO - Creating tables
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 27, in 
> args.func(args)
>   File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 897, 
> in initdb
> db_utils.initdb()
>   File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 
> 114, in initdb
> schema='airflow_ci'))
>   File "", line 4, in __init__
>   File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", 
> line 414, in _initialize_instance
> manager.dispatch.init_failure(self, args, kwargs)
>   File 
> "/usr/local/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
> line 66, in __exit__
> compat.reraise(exc_type, exc_value, exc_tb)
>   File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", 
> line 411, in _initialize_instance
> return manager.original_init(*mixed[1:], **kwargs)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 578, 
> in __init__
> self.extra = extra
>   File "", line 1, in __set__
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 639, 
> in set_extra
> fernet = get_fernet()
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 105, 
> in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
>   File "/usr/local/lib64/python2.7/site-packages/cryptography/fernet.py", 
> line 34, in __init__
> key = base64.urlsafe_b64decode(key)
>   File "/usr/lib64/python2.7/base64.py", line 119, in urlsafe_b64decode
> return b64decode(s.translate(_urlsafe_decode_translation))
>   File "/usr/lib64/python2.7/base64.py", line 78, in b64decode
> raise TypeError(msg)
> TypeError: Incorrect padding
> {panel}
> I also got an error when I try to add extras to a connection, if fernet_key 
> is empty in the config file. The error message was "Incorrect padding". Once 
> I provided a valid key generated with the instructions given 
> [here|http://airflow.readthedocs.io/en/latest/configuration.html?highlight=fernet#connections]
>  and restarted all of the airflow services it worked without any issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-2387) Flask 0.12.3 broke web component

2018-04-27 Thread Sergio B (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio B closed AIRFLOW-2387.
-
Resolution: Duplicate

> Flask 0.12.3 broke web component
> 
>
> Key: AIRFLOW-2387
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2387
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Sergio B
>Priority: Blocker
>
> Recent release of flask 0.12.3 broke the webinterface with this message. 
> Since requirement is >0.12 and < 0.13 any recent build will be broke
>  
> {code:java}
> AttributeError: 'Blueprint' object has no attribute 'json_encoder' flask{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2388) Add ability to configure paramiko client's banner_timeout in SSHHook

2018-04-27 Thread Avital (JIRA) 
Avital created AIRFLOW-2388:
---

 Summary: Add ability to configure paramiko client's banner_timeout 
in SSHHook
 Key: AIRFLOW-2388
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2388
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib, operators
Reporter: Avital


Right now, you can't increase the banner_timeout when connecting to a server 
through SSH. This is because `SSHHook` doesn't allow you to set the 
`banner_timeout` property on `paramiko.Client`'s `connect()` method, as 
documented here: [http://docs.paramiko.org/en/2.4/api/client.html] .

This negatively affects developers' ability to use SSH - for example, I'm 
getting DAG run fails as a result of `paramiko`'s inability to get an SSH 
banner, because I can't increase the number of seconds it can wait for the 
banner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2387) Flask 0.12.3 broke web component

2018-04-27 Thread Sergio B (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio B updated AIRFLOW-2387:
--
Priority: Blocker  (was: Major)

> Flask 0.12.3 broke web component
> 
>
> Key: AIRFLOW-2387
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2387
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Sergio B
>Priority: Blocker
>
> Recent release of flask 0.12.3 broke the webinterface with this message. 
> Since requirement is >0.12 and < 0.13 any recent build will be broke
>  
> {code:java}
> AttributeError: 'Blueprint' object has no attribute 'json_encoder' flask{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2387) Flask 0.12.3 broke web component

2018-04-27 Thread Sergio B (JIRA) 
Sergio B created AIRFLOW-2387:
-

 Summary: Flask 0.12.3 broke web component
 Key: AIRFLOW-2387
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2387
 Project: Apache Airflow
  Issue Type: Bug
  Components: webserver
Reporter: Sergio B


Recent release of flask 0.12.3 broke the webinterface with this message. Since 
requirement is >0.12 and < 0.13 any recent build will be broke

 
{code:java}
AttributeError: 'Blueprint' object has no attribute 'json_encoder' flask{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: closes apache/incubator-airflow#2827 *Closed for inactivity*

2018-04-27 Thread sanand 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 172b73c15 -> 67c0099e9


closes apache/incubator-airflow#2827 *Closed for inactivity*


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/67c0099e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/67c0099e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/67c0099e

Branch: refs/heads/master
Commit: 67c0099e97ae5d679f838d404bbbc52cefdf5201
Parents: 172b73c
Author: r39132 
Authored: Fri Apr 27 09:46:51 2018 -0700
Committer: r39132 
Committed: Fri Apr 27 09:46:51 2018 -0700

--

--

incubator-airflow git commit: closes apache/incubator-airflow#2586 *Closed for inactivity*

2018-04-27 Thread sanand 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master b92252158 -> 172b73c15


closes apache/incubator-airflow#2586 *Closed for inactivity*


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/172b73c1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/172b73c1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/172b73c1

Branch: refs/heads/master
Commit: 172b73c155f555c49c340275793c0c0a0b09bc24
Parents: b922521
Author: r39132 
Authored: Fri Apr 27 09:42:19 2018 -0700
Committer: r39132 
Committed: Fri Apr 27 09:42:19 2018 -0700

--

--

incubator-airflow git commit: closes apache/incubator-airflow#1933 *Closed for inactivity*

2018-04-27 Thread sanand 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 414a08e50 -> b92252158


closes apache/incubator-airflow#1933 *Closed for inactivity*


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b9225215
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b9225215
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b9225215

Branch: refs/heads/master
Commit: b9225215873251c061accd89349b82b99ad5d905
Parents: 414a08e
Author: r39132 
Authored: Fri Apr 27 09:39:13 2018 -0700
Committer: r39132 
Committed: Fri Apr 27 09:39:13 2018 -0700

--

--

[jira] [Commented] (AIRFLOW-1501) Google Cloud Storage delete operator

2018-04-27 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456584#comment-16456584
 ] 

Guillermo Rodríguez Cano commented on AIRFLOW-1501:
---

[~Fokko] This ticket is *not* resolved, please re-open. Functionality is 
available in the hook but not as an operator which is what is expected. 
I will be providing the corresponding operator soon unless 
[~yuu.ishik...@gmail.com] has something prepared, but the last comment is from 
August 2017.

> Google Cloud Storage delete operator
> 
>
> Key: AIRFLOW-1501
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1501
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, operators
>Reporter: Yu Ishikawa
>Assignee: Guillermo Rodríguez Cano
>Priority: Major
>
> h2. Goals
> - Implement a new feature to delete objects on Google Cloud Storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-1501) Google Cloud Storage delete operator

2018-04-27 Thread Anonymous (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anonymous reassigned AIRFLOW-1501:
--

Assignee: Guillermo Rodríguez Cano  (was: Yu Ishikawa)

> Google Cloud Storage delete operator
> 
>
> Key: AIRFLOW-1501
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1501
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, operators
>Reporter: Yu Ishikawa
>Assignee: Guillermo Rodríguez Cano
>Priority: Major
>
> h2. Goals
> - Implement a new feature to delete objects on Google Cloud Storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2381) Fix the Flaky ApiPasswordTests

2018-04-27 Thread Bolke de Bruin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-2381.
-
   Resolution: Fixed
Fix Version/s: 1.10.0

Issue resolved by pull request #3269
[https://github.com/apache/incubator-airflow/pull/3269]

> Fix the Flaky ApiPasswordTests
> --
>
> Key: AIRFLOW-2381
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2381
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the ApiPasswordTests fail because the dag is not available in the 
> database. I believe this is an issue with the different tests running 
> parallel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2381) Fix the Flaky ApiPasswordTests

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456542#comment-16456542
 ] 

ASF subversion and git services commented on AIRFLOW-2381:
--

Commit 4d64ad4928f0188f7532936e8da6612f5ec7170d in incubator-airflow's branch 
refs/heads/v1-10-test from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4d64ad4 ]

[AIRFLOW-2381] Fix the flaky ApiPasswordTests test

This test is in conflict with different tests
running in parallel
By calling a simple overview page, the behaviour
of checking the
password is still checked, but isn't dependent on
a specific dag
being present in the database

Closes #3269 from Fokko/AIRFLOW-2381

(cherry picked from commit 414a08e505e2ae86f4272bfc8b0d28d66ef6ab6a)
Signed-off-by: Bolke de Bruin 


> Fix the Flaky ApiPasswordTests
> --
>
> Key: AIRFLOW-2381
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2381
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the ApiPasswordTests fail because the dag is not available in the 
> database. I believe this is an issue with the different tests running 
> parallel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2381] Fix the flaky ApiPasswordTests test

2018-04-27 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-10-test cb264e940 -> 4d64ad492


[AIRFLOW-2381] Fix the flaky ApiPasswordTests test

This test is in conflict with different tests
running in parallel
By calling a simple overview page, the behaviour
of checking the
password is still checked, but isn't dependent on
a specific dag
being present in the database

Closes #3269 from Fokko/AIRFLOW-2381

(cherry picked from commit 414a08e505e2ae86f4272bfc8b0d28d66ef6ab6a)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/4d64ad49
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/4d64ad49
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/4d64ad49

Branch: refs/heads/v1-10-test
Commit: 4d64ad4928f0188f7532936e8da6612f5ec7170d
Parents: cb264e9
Author: Fokko Driesprong 
Authored: Fri Apr 27 16:43:40 2018 +0200
Committer: Bolke de Bruin 
Committed: Fri Apr 27 16:43:55 2018 +0200

--
 .../api/experimental/test_password_endpoints.py  | 19 +--
 1 file changed, 5 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4d64ad49/tests/www/api/experimental/test_password_endpoints.py
--
diff --git a/tests/www/api/experimental/test_password_endpoints.py 
b/tests/www/api/experimental/test_password_endpoints.py
index ecddff1..8131ff5 100644
--- a/tests/www/api/experimental/test_password_endpoints.py
+++ b/tests/www/api/experimental/test_password_endpoints.py
@@ -7,9 +7,9 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -56,24 +56,15 @@ class ApiPasswordTests(unittest.TestCase):
 
 def test_authorized(self):
 with self.app.test_client() as c:
-url_template = '/api/experimental/dags/{}/dag_runs'
-response = c.post(
-url_template.format('example_bash_operator'),
-data=json.dumps(dict(run_id='my_run' + 
datetime.now().isoformat())),
-content_type="application/json",
+response = c.get(
+'/api/experimental/pools',
 headers={'Authorization': 'Basic aGVsbG86d29ybGQ='}  # 
hello:world
 )
 self.assertEqual(200, response.status_code)
 
 def test_unauthorized(self):
 with self.app.test_client() as c:
-url_template = '/api/experimental/dags/{}/dag_runs'
-response = c.post(
-url_template.format('example_bash_operator'),
-data=json.dumps(dict(run_id='my_run' + 
datetime.now().isoformat())),
-content_type="application/json"
-)
-
+response = c.get('/api/experimental/pools')
 self.assertEqual(401, response.status_code)
 
 def tearDown(self):

[jira] [Commented] (AIRFLOW-2381) Fix the Flaky ApiPasswordTests

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456539#comment-16456539
 ] 

ASF subversion and git services commented on AIRFLOW-2381:
--

Commit 414a08e505e2ae86f4272bfc8b0d28d66ef6ab6a in incubator-airflow's branch 
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=414a08e ]

[AIRFLOW-2381] Fix the flaky ApiPasswordTests test

This test is in conflict with different tests
running in parallel
By calling a simple overview page, the behaviour
of checking the
password is still checked, but isn't dependent on
a specific dag
being present in the database

Closes #3269 from Fokko/AIRFLOW-2381


> Fix the Flaky ApiPasswordTests
> --
>
> Key: AIRFLOW-2381
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2381
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Priority: Major
>
> Currently the ApiPasswordTests fail because the dag is not available in the 
> database. I believe this is an issue with the different tests running 
> parallel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2381) Fix the Flaky ApiPasswordTests

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456540#comment-16456540
 ] 

ASF subversion and git services commented on AIRFLOW-2381:
--

Commit 414a08e505e2ae86f4272bfc8b0d28d66ef6ab6a in incubator-airflow's branch 
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=414a08e ]

[AIRFLOW-2381] Fix the flaky ApiPasswordTests test

This test is in conflict with different tests
running in parallel
By calling a simple overview page, the behaviour
of checking the
password is still checked, but isn't dependent on
a specific dag
being present in the database

Closes #3269 from Fokko/AIRFLOW-2381


> Fix the Flaky ApiPasswordTests
> --
>
> Key: AIRFLOW-2381
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2381
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Priority: Major
>
> Currently the ApiPasswordTests fail because the dag is not available in the 
> database. I believe this is an issue with the different tests running 
> parallel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2381] Fix the flaky ApiPasswordTests test

2018-04-27 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 801fe7dbd -> 414a08e50


[AIRFLOW-2381] Fix the flaky ApiPasswordTests test

This test is in conflict with different tests
running in parallel
By calling a simple overview page, the behaviour
of checking the
password is still checked, but isn't dependent on
a specific dag
being present in the database

Closes #3269 from Fokko/AIRFLOW-2381


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/414a08e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/414a08e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/414a08e5

Branch: refs/heads/master
Commit: 414a08e505e2ae86f4272bfc8b0d28d66ef6ab6a
Parents: 801fe7d
Author: Fokko Driesprong 
Authored: Fri Apr 27 16:43:40 2018 +0200
Committer: Bolke de Bruin 
Committed: Fri Apr 27 16:43:40 2018 +0200

--
 .../api/experimental/test_password_endpoints.py  | 19 +--
 1 file changed, 5 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/414a08e5/tests/www/api/experimental/test_password_endpoints.py
--
diff --git a/tests/www/api/experimental/test_password_endpoints.py 
b/tests/www/api/experimental/test_password_endpoints.py
index ecddff1..8131ff5 100644
--- a/tests/www/api/experimental/test_password_endpoints.py
+++ b/tests/www/api/experimental/test_password_endpoints.py
@@ -7,9 +7,9 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -56,24 +56,15 @@ class ApiPasswordTests(unittest.TestCase):
 
 def test_authorized(self):
 with self.app.test_client() as c:
-url_template = '/api/experimental/dags/{}/dag_runs'
-response = c.post(
-url_template.format('example_bash_operator'),
-data=json.dumps(dict(run_id='my_run' + 
datetime.now().isoformat())),
-content_type="application/json",
+response = c.get(
+'/api/experimental/pools',
 headers={'Authorization': 'Basic aGVsbG86d29ybGQ='}  # 
hello:world
 )
 self.assertEqual(200, response.status_code)
 
 def test_unauthorized(self):
 with self.app.test_client() as c:
-url_template = '/api/experimental/dags/{}/dag_runs'
-response = c.post(
-url_template.format('example_bash_operator'),
-data=json.dumps(dict(run_id='my_run' + 
datetime.now().isoformat())),
-content_type="application/json"
-)
-
+response = c.get('/api/experimental/pools')
 self.assertEqual(401, response.status_code)
 
 def tearDown(self):

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: 
When I run an airflow backfill using the airflow cli (i.e. airflow backfill -s 
somedate -e someotherdate dag_id), if the start date of the backfill is before 
the start_date of the dag, then airflow ignores depends_on_past and runs tasks 
in parallel for different days that have depends_on_past set to true. If the 
end date of the backfill is after the start_date of the dag then the backfill 
will begin respecting depends_on_past once it reaches the start_date of the 
dag. Seems like a bug.

LocalExecutor
airflow 1.9
python 2.7.9

  was:
When I run an airflow backfill using the airflow cli (i.e. airflow backfill -s 
somedate -e someotherdate dag_id), if the start date of the backfill is before 
the start_date of the dag, then airflow ignores depends_on_past and runs tasks 
in parallel for different days that have depends_on_past set to true. If the 
end date of the backfill is after the start_date of the dag then the backfill 
will begin respecting depends_on_past once it reaches the start_date of the 
dag. Seems like a bug.

airflow 1.9
LocalExecutor
python 2.7.9


> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli (i.e. airflow backfill 
> -s somedate -e someotherdate dag_id), if the start date of the backfill is 
> before the start_date of the dag, then airflow ignores depends_on_past and 
> runs tasks in parallel for different days that have depends_on_past set to 
> true. If the end date of the backfill is after the start_date of the dag then 
> the backfill will begin respecting depends_on_past once it reaches the 
> start_date of the dag. Seems like a bug.
> LocalExecutor
> airflow 1.9
> python 2.7.9



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: 
When I run an airflow backfill using the airflow cli (i.e. airflow backfill -s 
somedate -e someotherdate dag_id), if the start date of the backfill is before 
the start_date of the dag, then airflow ignores depends_on_past and runs tasks 
in parallel for different days that have depends_on_past set to true. If the 
end date of the backfill is after the start_date of the dag then the backfill 
will begin respecting depends_on_past once it reaches the start_date of the 
dag. Seems like a bug.

airflow 1.9
python 2.7.9
amazonlinux

  was:When I run an airflow backfill using the airflow cli (i.e. airflow 
backfill -s somedate -e someotherdate dag_id), if the start date of the 
backfill is before the start_date of the dag, then airflow ignores 
depends_on_past and runs tasks in parallel for different days that have 
depends_on_past set to true. If the end date of the backfill is after the 
start_date of the dag then the backfill will begin respecting depends_on_past 
once it reaches the start_date of the dag. Seems like a bug.


> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli (i.e. airflow backfill 
> -s somedate -e someotherdate dag_id), if the start date of the backfill is 
> before the start_date of the dag, then airflow ignores depends_on_past and 
> runs tasks in parallel for different days that have depends_on_past set to 
> true. If the end date of the backfill is after the start_date of the dag then 
> the backfill will begin respecting depends_on_past once it reaches the 
> start_date of the dag. Seems like a bug.
> airflow 1.9
> python 2.7.9
> amazonlinux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: 
When I run an airflow backfill using the airflow cli (i.e. airflow backfill -s 
somedate -e someotherdate dag_id), if the start date of the backfill is before 
the start_date of the dag, then airflow ignores depends_on_past and runs tasks 
in parallel for different days that have depends_on_past set to true. If the 
end date of the backfill is after the start_date of the dag then the backfill 
will begin respecting depends_on_past once it reaches the start_date of the 
dag. Seems like a bug.

airflow 1.9
LocalExecutor
python 2.7.9

  was:
When I run an airflow backfill using the airflow cli (i.e. airflow backfill -s 
somedate -e someotherdate dag_id), if the start date of the backfill is before 
the start_date of the dag, then airflow ignores depends_on_past and runs tasks 
in parallel for different days that have depends_on_past set to true. If the 
end date of the backfill is after the start_date of the dag then the backfill 
will begin respecting depends_on_past once it reaches the start_date of the 
dag. Seems like a bug.

airflow 1.9
python 2.7.9
amazonlinux


> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli (i.e. airflow backfill 
> -s somedate -e someotherdate dag_id), if the start date of the backfill is 
> before the start_date of the dag, then airflow ignores depends_on_past and 
> runs tasks in parallel for different days that have depends_on_past set to 
> true. If the end date of the backfill is after the start_date of the dag then 
> the backfill will begin respecting depends_on_past once it reaches the 
> start_date of the dag. Seems like a bug.
> airflow 1.9
> LocalExecutor
> python 2.7.9



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: When I run an airflow backfill using the airflow cli (i.e. 
airflow backfill -s somedate -e someotherdate dag_id), if the start date of the 
backfill is before the start_date of the dag, then airflow ignores 
depends_on_past and runs tasks in parallel for different days that have 
depends_on_past set to true. If the end date of the backfill is after the 
start_date of the dag then the backfill will begin respecting depends_on_past 
once it reaches the start_date of the dag. Seems like a bug.  (was: When I run 
an airflow backfill using the airflow cli if the start date of the backfill is 
before the start_date of the dag, then airflow ignores depends_on_past and runs 
tasks in parallel for different days that have depends_on_past set to true. If 
the end date of the backfill is after the start_date of the dag then the 
backfill will begin respecting depends_on_past once it reaches the start_date 
of the dag. Seems like a bug.)

> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli (i.e. airflow backfill 
> -s somedate -e someotherdate dag_id), if the start date of the backfill is 
> before the start_date of the dag, then airflow ignores depends_on_past and 
> runs tasks in parallel for different days that have depends_on_past set to 
> true. If the end date of the backfill is after the start_date of the dag then 
> the backfill will begin respecting depends_on_past once it reaches the 
> start_date of the dag. Seems like a bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: When I run an airflow backfill using the airflow cli if the 
start date of the backfill is before the start_date of the dag, then airflow 
ignores depends_on_past and runs tasks in parallel for different days that have 
depends_on_past set to true. If the end date of the backfill is after the 
start_date of the dag then the backfill will begin respecting depends_on_past 
once it reaches the start_date of the dag. Seems like a bug.  (was: When I run 
an airflow backfill using the airflow cli if the start date of the backfill is 
before the start date of the dag, then airflow ignores depends_on_past and runs 
tasks in parallel for different days that have depends_on_past set to true. If 
the end date of the backfill is after the start_date of the dag then the 
backfill will begin respecting depends_on_past once it reaches the start_date 
of the dag. Seems like a bug.)

> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli if the start date of the 
> backfill is before the start_date of the dag, then airflow ignores 
> depends_on_past and runs tasks in parallel for different days that have 
> depends_on_past set to true. If the end date of the backfill is after the 
> start_date of the dag then the backfill will begin respecting depends_on_past 
> once it reaches the start_date of the dag. Seems like a bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sulphide updated AIRFLOW-2386:
--
Description: When I run an airflow backfill using the airflow cli if the 
start date of the backfill is before the start date of the dag, then airflow 
ignores depends_on_past and runs tasks in parallel for different days that have 
depends_on_past set to true. If the end date of the backfill is after the 
start_date of the dag then the backfill will begin respecting depends_on_past 
once it reaches the start_date of the dag. Seems like a bug.

> airflow backfill ignores depends_on_past when backfilling before start_date
> ---
>
> Key: AIRFLOW-2386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: sulphide
>Priority: Major
>
> When I run an airflow backfill using the airflow cli if the start date of the 
> backfill is before the start date of the dag, then airflow ignores 
> depends_on_past and runs tasks in parallel for different days that have 
> depends_on_past set to true. If the end date of the backfill is after the 
> start_date of the dag then the backfill will begin respecting depends_on_past 
> once it reaches the start_date of the dag. Seems like a bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2386) airflow backfill ignores depends_on_past when backfilling before start_date

2018-04-27 Thread sulphide (JIRA) 
sulphide created AIRFLOW-2386:
-

 Summary: airflow backfill ignores depends_on_past when backfilling 
before start_date
 Key: AIRFLOW-2386
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2386
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: sulphide






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2331) Add support for initialization action timeout on dataproc cluster creation.

2018-04-27 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cristòfol Torrens updated AIRFLOW-2331:
---
Fix Version/s: (was: 2.0.0)
   1.10.0

> Add support for initialization action timeout on dataproc cluster creation.
> ---
>
> Key: AIRFLOW-2331
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2331
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Reporter: Cristòfol Torrens
>Assignee: Cristòfol Torrens
>Priority: Minor
>  Labels: contrib, dataproc, operator
> Fix For: 1.10.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Add support to customize timeout for initialization scripts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2342) DAG in running state but tasks not running

2018-04-27 Thread Sathyaprakash Govindasamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455980#comment-16455980
 ] 

Sathyaprakash Govindasamy commented on AIRFLOW-2342:


I see that error occurs when _SqlSensor_ is tried to be imported in 
_daily_building_blocks_main_ dag

File "/home/airflow_sa/dags/daily_building_blocks_main.py", line 14, in 
 from airflow.operators.sensors import SqlSensor

You can run interactive python shell and run this below line to see whether it 
throws any error
{code:java}
from airflow.operators.sensors import SqlSensor{code}
This error is caused by _snakbite_ import in hdfs_hook (which is internally 
called in _sensors_ module)

File "/usr/local/lib/python2.7/site-packages/airflow/hooks/hdfs_hook.py", line 
20, in  from snakebite.client import Client, HAClient, Namenode, 
AutoConfigClient

But I see that in 1.8, this import is within try block. So I am not sure why 
this import statement is getting filed. 

[https://github.com/apache/incubator-airflow/blob/4e370ffc9ab579aa75de0fa8704c96683b9b963e/airflow/hooks/hdfs_hook.py#L20]

You can inspect the file 
*/usr/local/lib/python2.7/site-packages/airflow/hooks/hdfs_hook.py* in your 
system to see whether this import is really in try block

> DAG in running state but tasks not running
> --
>
> Key: AIRFLOW-2342
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2342
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
>Affects Versions: Airflow 1.8
> Environment: Redhat
>Reporter: chidrup jhanjhari
>Priority: Major
> Attachments: job1.py.log
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Hi, We are on Airflow 1.8.0. We have airflow production environment running 
> well since 8 months. There has been no change on configuration etc. The issue 
> is since 2 days DAGs are showing in running state but the tasks are not 
> getting triggered. After the default start task, the DAG run is not moving to 
> the next task. Attached is the scheduler throwing following error:
> 2018-04-19 01:32:22,586] \{jobs.py:354} DagFileProcessor17 ERROR - Got an 
> exception! Propagating...
> Traceback (most recent call last):
>   Any help will be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2385) Airflow task is not stopped when execution timeout gets triggered

2018-04-27 Thread Yohei Onishi (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455799#comment-16455799
 ] 

Yohei Onishi commented on AIRFLOW-2385:
---

Yes, this job runs Spark job. I added on_kill method and stop Spark session.  I 
will see if it works. Any advices are highly welcomed.
{code:java}
def execute(self, context):
  self.spark = self.create_spark_session()
  // run some queries

def create_spark_session(self):
  return SparkSession.builder. \
appName(self.app_name). \
config('spark.master', self.spark_master). \
config('spark.executor.cores', self.spark_executor_cores). \
config('spark.executor.memory', self.spark_executor_memory). \
config('spark.executor.instances', self.spark_num_executors). \
config('spark.yarn.queue', self.yarn_queue). \
enableHiveSupport(). \
getOrCreate()

def on_kill(self):
  if (self.spark):
self.spark.stop()
{code}

> Airflow task is not stopped when execution timeout gets triggered
> -
>
> Key: AIRFLOW-2385
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2385
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.9.0
>Reporter: Yohei Onishi
>Priority: Major
>
> I have my own custom operator extends BaseOperator as follows. I tried to 
> kill a task if the task runs for more than 30 minutes. timeout seems to be 
> triggered according to a log but the task still continued.
> Am I missing something? I checked the official document but do not know what 
> is wrong.[https://airflow.apache.org/code.html#baseoperator]
> My operator is like as follows.
> {code:java}
> class MyOperator(BaseOperator):
>   @apply_defaults
>   def __init__(
> self,
> some_parameters_here,
> *args,
> **kwargs):
> super(MyOperator, self).__init__(*args, **kwargs)
> # some initialization here
>   def execute(self, context):
> # some code here
> {code}
>  
> {{}}My task is like as follows.
> {code:java}
> t = MyOperator(
>   task_id='task',
>   dag=scheduled_dag,
>   execution_timeout=timedelta(minutes=30)
> {code}
>  
> I found this error but the task continued.
> {code:java}
> [2018-04-12 03:30:28,353] {base_task_runner.py:98} INFO - Subtask: [Stage 
> 6:==(1380 + -160) / 
> 1224][2018-04- 12 03:30:28,353] {timeout.py:36} ERROR - Process timed out
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)