[jira] [Closed] (AIRFLOW-386) limit github enterprise auth user scope

2016-08-01 Thread Siddharth Anand (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Anand closed AIRFLOW-386.
---
Resolution: Fixed

> limit github enterprise auth user scope
> ---
>
> Key: AIRFLOW-386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-386
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Michael Lyons
>Assignee: Michael Lyons
>  Labels: security
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current github enterprise auth code requests read/write access to a user 
> profile which is probably not required for simple login?
> The change is pretty straight forward:
> {code}
> request_token_params={'scope': 'user,read:org'},
> {code}
> to 
> {code}
> request_token_params={'scope': 'user:email,read:org'},
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-386) limit github enterprise auth user scope

2016-08-01 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402821#comment-15402821
 ] 

ASF subversion and git services commented on AIRFLOW-386:
-

Commit a1c4cd92d536f7bf6617d85df4d197d8adf09d3d in incubator-airflow's branch 
refs/heads/master from michlyon
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a1c4cd9 ]

limit scope to user email only AIRFLOW-386


> limit github enterprise auth user scope
> ---
>
> Key: AIRFLOW-386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-386
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Michael Lyons
>Assignee: Michael Lyons
>  Labels: security
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current github enterprise auth code requests read/write access to a user 
> profile which is probably not required for simple login?
> The change is pretty straight forward:
> {code}
> request_token_params={'scope': 'user,read:org'},
> {code}
> to 
> {code}
> request_token_params={'scope': 'user:email,read:org'},
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-airflow git commit: Merge pull request #1700 from mylons/limit_github_enterprise_scope

2016-08-01 Thread sanand 
Merge pull request #1700 from mylons/limit_github_enterprise_scope


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3cef8ed2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3cef8ed2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3cef8ed2

Branch: refs/heads/master
Commit: 3cef8ed28b5bd18f347cf84b4d1a64bc03b39143
Parents: eb989dc a1c4cd9
Author: Sid Anand 
Authored: Mon Aug 1 14:08:34 2016 -0700
Committer: Sid Anand 
Committed: Mon Aug 1 14:08:34 2016 -0700

--
 airflow/contrib/auth/backends/github_enterprise_auth.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-airflow git commit: limit scope to user email only AIRFLOW-386

2016-08-01 Thread sanand 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master eb989dcb5 -> 3cef8ed28


limit scope to user email only AIRFLOW-386


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a1c4cd92
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a1c4cd92
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a1c4cd92

Branch: refs/heads/master
Commit: a1c4cd92d536f7bf6617d85df4d197d8adf09d3d
Parents: eb989dc
Author: michlyon 
Authored: Mon Aug 1 10:16:22 2016 -0700
Committer: michlyon 
Committed: Mon Aug 1 10:16:22 2016 -0700

--
 airflow/contrib/auth/backends/github_enterprise_auth.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a1c4cd92/airflow/contrib/auth/backends/github_enterprise_auth.py
--
diff --git a/airflow/contrib/auth/backends/github_enterprise_auth.py 
b/airflow/contrib/auth/backends/github_enterprise_auth.py
index c3eb96b..f9d6426 100644
--- a/airflow/contrib/auth/backends/github_enterprise_auth.py
+++ b/airflow/contrib/auth/backends/github_enterprise_auth.py
@@ -101,7 +101,7 @@ class GHEAuthBackend(object):
 consumer_key=get_config_param('client_id'),
 consumer_secret=get_config_param('client_secret'),
 # need read:org to get team member list
-request_token_params={'scope': 'user,read:org'},
+request_token_params={'scope': 'user:email,read:org'},
 base_url=self.ghe_host,
 request_token_url=None,
 access_token_method='POST',

[jira] [Commented] (AIRFLOW-386) limit github enterprise auth user scope

2016-08-01 Thread Michael Lyons (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402689#comment-15402689
 ] 

Michael Lyons commented on AIRFLOW-386:
---

[~criccomini] was trying to figure out the testing setup, but have to throw in 
the towel for now. submitting PR

> limit github enterprise auth user scope
> ---
>
> Key: AIRFLOW-386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-386
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Michael Lyons
>Assignee: Michael Lyons
>  Labels: security
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current github enterprise auth code requests read/write access to a user 
> profile which is probably not required for simple login?
> The change is pretty straight forward:
> {code}
> request_token_params={'scope': 'user,read:org'},
> {code}
> to 
> {code}
> request_token_params={'scope': 'user:email,read:org'},
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-386) limit github enterprise auth user scope

2016-08-01 Thread Chris Riccomini (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402603#comment-15402603
 ] 

Chris Riccomini commented on AIRFLOW-386:
-

Makes sense. Want to send a PR?

> limit github enterprise auth user scope
> ---
>
> Key: AIRFLOW-386
> URL: https://issues.apache.org/jira/browse/AIRFLOW-386
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Michael Lyons
>Assignee: Michael Lyons
>  Labels: security
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current github enterprise auth code requests read/write access to a user 
> profile which is probably not required for simple login?
> The change is pretty straight forward:
> {code}
> request_token_params={'scope': 'user,read:org'},
> {code}
> to 
> {code}
> request_token_params={'scope': 'user:email,read:org'},
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-386) limit github enterprise auth user scope

2016-08-01 Thread Michael Lyons (JIRA) 
Michael Lyons created AIRFLOW-386:
-

 Summary: limit github enterprise auth user scope
 Key: AIRFLOW-386
 URL: https://issues.apache.org/jira/browse/AIRFLOW-386
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Michael Lyons
Assignee: Michael Lyons


The current github enterprise auth code requests read/write access to a user 
profile which is probably not required for simple login?

The change is pretty straight forward:
{code}
request_token_params={'scope': 'user,read:org'},
{code}
to 
{code}
request_token_params={'scope': 'user:email,read:org'},
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-383) Fix example_qubole_operator DAG

2016-08-01 Thread Chris Riccomini (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-383.
---
Resolution: Fixed

> Fix example_qubole_operator DAG
> ---
>
> Key: AIRFLOW-383
> URL: https://issues.apache.org/jira/browse/AIRFLOW-383
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Yogesh Garg
>Assignee: Yogesh Garg
>Priority: Minor
> Fix For: Airflow 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-383) Fix example_qubole_operator DAG

2016-08-01 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402159#comment-15402159
 ] 

ASF subversion and git services commented on AIRFLOW-383:
-

Commit eb989dcb54387c218683de9dff0a12950dce147b in incubator-airflow's branch 
refs/heads/master from [~yogeshg]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=eb989dc ]

[AIRFLOW-383] Cleanup example qubole operator dag

Closes #1698 from yogesh2021/AIRFLOW-383


> Fix example_qubole_operator DAG
> ---
>
> Key: AIRFLOW-383
> URL: https://issues.apache.org/jira/browse/AIRFLOW-383
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Yogesh Garg
>Assignee: Yogesh Garg
>Priority: Minor
> Fix For: Airflow 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-airflow git commit: [AIRFLOW-383] Cleanup example qubole operator dag

2016-08-01 Thread criccomini 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master fdb7e9491 -> eb989dcb5


[AIRFLOW-383] Cleanup example qubole operator dag

Closes #1698 from yogesh2021/AIRFLOW-383


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/eb989dcb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/eb989dcb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/eb989dcb

Branch: refs/heads/master
Commit: eb989dcb54387c218683de9dff0a12950dce147b
Parents: fdb7e94
Author: Yogesh Garg 
Authored: Mon Aug 1 07:49:54 2016 -0700
Committer: Chris Riccomini 
Committed: Mon Aug 1 07:49:54 2016 -0700

--
 .../example_dags/example_qubole_operator.py | 23 +---
 1 file changed, 10 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/eb989dcb/airflow/contrib/example_dags/example_qubole_operator.py
--
diff --git a/airflow/contrib/example_dags/example_qubole_operator.py 
b/airflow/contrib/example_dags/example_qubole_operator.py
index 63cccd3..ff363e6 100644
--- a/airflow/contrib/example_dags/example_qubole_operator.py
+++ b/airflow/contrib/example_dags/example_qubole_operator.py
@@ -32,7 +32,9 @@ default_args = {
 'email_on_retry': False
 }
 
-dag = DAG('example_qubole_operator', default_args=default_args)
+# NOTE:: This is only an example DAG to highlight usage of QuboleOperator in 
various scenarios,
+# some of the tasks may or may not work based on your QDS account setup
+dag = DAG('example_qubole_operator', default_args=default_args, 
schedule_interval='@daily')
 
 def compare_result(ds, **kwargs):
 ti = kwargs['ti']
@@ -45,14 +47,15 @@ t1 = QuboleOperator(
 command_type='hivecmd',
 query='show tables',
 cluster_label='default',
-fetch_logs=True,
-tags='aiflow_example_run',
+fetch_logs=True, # If true, will fetch qubole command logs and concatenate 
them into corresponding airflow task logs
+tags='aiflow_example_run',  # To attach tags to qubole command, auto 
attach 3 tags - dag_id, task_id, run_id
+qubole_conn_id='qubole_default',  # Connection id to submit commands 
inside QDS, if not set "qubole_default" is used
 dag=dag)
 
 t2 = QuboleOperator(
 task_id='hive_s3_location',
 command_type="hivecmd",
-script_location="s3n://dev.canopydata.com/airflow/show_table.hql",
+script_location="s3n://public-qubole/qbol-library/scripts/show_table.hql",
 notfiy=True,
 tags=['tag1', 'tag2'],
 trigger_rule="all_done",
@@ -76,14 +79,12 @@ branching = BranchPythonOperator(
 dag=dag)
 branching.set_upstream(t3)
 
-
 join = DummyOperator(
 task_id='join',
 trigger_rule='one_success',
 dag=dag
 )
 
-
 t4 = QuboleOperator(
 task_id='hadoop_jar_cmd',
 command_type='hadoopcmd',
@@ -95,7 +96,7 @@ t4 = QuboleOperator(
 t5 = QuboleOperator(
 task_id='pig_cmd',
 command_type="pigcmd",
-
script_location="s3://paid-qubole/PigAPIDemo/scripts/script1-hadoop-s3-small.pig",
+
script_location="s3://public-qubole/qbol-library/scripts/script1-hadoop-s3-small.pig",
 parameters="key1=value1 key2=value2",
 trigger_rule="all_done",
 dag=dag)
@@ -104,7 +105,6 @@ t4.set_upstream(branching)
 t5.set_upstream(t4)
 t5.set_downstream(join)
 
-
 t6 = QuboleOperator(
 task_id='presto_cmd',
 command_type='prestocmd',
@@ -114,7 +114,7 @@ t6 = QuboleOperator(
 t7 = QuboleOperator(
 task_id='shell_cmd',
 command_type="shellcmd",
-script_location="s3://paid-qubole/ShellDemo/data/excite-small.sh",
+script_location="s3://public-qubole/qbol-library/scripts/shellx.sh",
 parameters="param1 param2",
 trigger_rule="all_done",
 dag=dag)
@@ -123,7 +123,6 @@ t6.set_upstream(branching)
 t7.set_upstream(t6)
 t7.set_downstream(join)
 
-
 t8 = QuboleOperator(
 task_id='db_query',
 command_type='dbtapquerycmd',
@@ -146,7 +145,6 @@ t8.set_upstream(branching)
 t9.set_upstream(t8)
 t9.set_downstream(join)
 
-
 t10 = QuboleOperator(
 task_id='db_import',
 command_type='dbimportcmd',
@@ -186,7 +184,7 @@ t11 = QuboleOperator(
 task_id='spark_cmd',
 command_type="sparkcmd",
 program=prog,
-language='python',
+language='scala',
 arguments='--class SparkPi',
 tags='aiflow_example_run',
 dag=dag)
@@ -194,4 +192,3 @@ t11 = QuboleOperator(
 t11.set_upstream(branching)
 t11.set_downstream(t10)
 t10.set_downstream(join)
-

[jira] [Updated] (AIRFLOW-194) Task hangs in up_for_retry state for very long

2016-08-01 Thread Michal TOMA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal TOMA updated AIRFLOW-194:

Attachment: screenshot-2.png

> Task hangs in up_for_retry state for very long
> --
>
> Key: AIRFLOW-194
> URL: https://issues.apache.org/jira/browse/AIRFLOW-194
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.0
> Environment: Airflow 1.7.0 on RHEL 7 and OpenSuse 13.2
>Reporter: Michal TOMA
>Assignee: Siddharth Anand
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> I can observe this problem on 2 separate Airflow installations.
> The symptoms are:
> - One (and only one) task stays in up_for_retry state even when the last of 
> the retries finished with an OK stays.
> - It is yellow in the tree view.
> - The execution somehow resumes several hours later automatically
> - It seems (not a certitude) related to a mode when the task execution is 
> "lagging" behind normal execution.
> Here is an example of a task that should run every hour "0 * * * *":
> Current date : 2016-05-30T15:31:00+0200
> - Run 1 --
> Run ID: 2016-05-05T21:00:00
> Task start: 2015-05-30T07:38:XX.XXX
> Task end: 2015-05-30T08:23:XX.XXX
> Marked as success
> - Run 2 --
> Run ID: 2016-05-05T22:00:00
> Task start: 2015-05-30T11:10:XX.XXX
> Task end: 2015-05-30T11:56:XX.XXX
> Marked as success
> - Run 3 --
> Run ID: 2016-05-05T23:00:00
> Task start: 2015-05-30T11:56:XX.XXX
> Task end: 2015-05-30T12:41:XX.XXX
> Marked as success
> - Run 4 --
> Run ID: 2016-05-06T00:00:00
> Task start: 2015-05-30T15:12:XX.XXX
> Task end: (Still running now)
> Marked as running
> There are nearly 2 hours between Run-1 and Run-2, and nearly 2 hours as well 
> between Run-3 and Run-4.
> Only Run-3 starts immediately after the end of Run-2 what is the expected 
> behavior as the Runs are very late on schedule (Run ID is 2016-05-06 while we 
> are on 2016-05-30)
> This is a high priority issue for our setup. I could try to dig more in depth 
> into this problem but I have no idea where to look to debug this issue.
> Any pointers would be more than welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-352) filter_by_owner is not working when use ldap authentication

2016-08-01 Thread peter pang (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401741#comment-15401741
 ] 

peter pang commented on AIRFLOW-352:


These modifications affect all the authentication methods. 

for example,
password_auth.py --> class PasswordUser(models.User): 
must added the same method:

def get_username(self):
return self.user.username



> filter_by_owner is not working when use ldap authentication
> ---
>
> Key: AIRFLOW-352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-352
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, security, webserver
>Affects Versions: Airflow 1.7.1.3
> Environment: ubuntu 14.04 LTS ,  ldap without encryption 
>Reporter: peter pang
>  Labels: security
>
> I set airflow.cfg as follows:
> {noformat}
> [webserver]
> filter_by_owner = True
> authenticate = TRUE
> auth_backend = airflow.contrib.auth.backends.ldap_auth
> [ldap]
> uri = ldap://xx.xx.xx.xx
> user_filter = objectClass=*
> user_name_attr = uid
> superuser_filter = 
> memberOf=CN=airflow-super-users,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> data_profiler_filter = 
> memberOf=CN=airflow-data-profilers,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> bind_user = cn=admin,dc=example,dc=com
> bind_password = secret
> basedn = dc=example,dc=com
> cacert = /etc/ca/ldap_ca.crt
> search_scope=SUBTREE
> {noformat}
> then I run the webUI , and I can login with superuser and data_profiler user. 
> But after login with data profiler user, entered the data profiler user home 
> view , there's no dags listed with the same dag owner. It seems the  
> filter_by_owner setting is not working.
> Debug into the views.py --> class HomeView(AdminIndexView):
> {color:red}current_user.username{color} always get{color:red} "None"{color}. 
> It seems we can't get username directly.
> so , continue debug into the ldap_auth.py --> class LdapUser(models.User):
> I added a method to return username   
> {code}
>  def get_username(self):
> return self.user.username
> {code}
> then back to view.py  , replace 'current_user.username' to 
> {color:red}'current_user.get_username()'{color} , the user filter can work 
> now!
> I don't know exactly why, but the modification can work...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)