Re: [jupyter] Getting Started with Jupyter on a Kerberized cluster

2016-10-17 Thread Miguel Ángel González Fernández 
Hi  Gerg,

You meant that with Jupyter Hub the integration with Kerberos is already 
working ?

Any tip or guide?

Thank you.

MIguelÁngel

El jueves, 13 de octubre de 2016, 14:24:51 (UTC+2), Gerg escribió:
>
> Hi Tristan-
>
> I completely agree with you that it is straight forward when you run 
> notebook themselves. But with little complications in getting the jupyter 
> hub to work in kerborized environment, i would say jupyter hub works 
> awesome (Kudos goes to jupyter engineers Min and Carol to name few), after 
> the setup it reduces the work on the admin side and new users can easily 
> get started. Jupyter hub takes care of all the ticket and negotiations that 
> happen while talking.
>
> Cheers,
> Gerg
>
> On Thursday, October 13, 2016 at 3:38:47 AM UTC-4, Tristan Zajonc wrote:
>>
>> Gerg,
>>
>> If users run the notebook themselves, as they would on their laptop, this 
>> should be relatively straightforward.  Users can just run kinit and their 
>> keytabs should get picked up by most Hadoop tools like Spark.  In a 
>> multi-user or containerized setup it may be more complicated.
>>
>> Tristan
>>
>> On Tuesday, October 4, 2016 at 2:41:44 AM UTC-7, Hani1814 wrote:
>>>
>>> Hi Greg,
>>>
>>> Is it possible to share the set-up?
>>>
>>> On Monday, 27 June 2016 12:46:26 UTC+1, Gerg wrote:

 Hi Deepak-

 We are not there yet, but information that min provided will be helpful 
 in setting up HUB.
 We will update you once we have something

 Thanks

 On Friday, June 24, 2016 at 6:38:25 AM UTC-4, Deepak Subhramanian wrote:
>
> We are also trying to get Jupyter working with Kerberized cluster. Did 
> anyone managed to get it working ?
>
> On Thursday, 16 June 2016 05:58:17 UTC-7, Min RK wrote:
>>
>> I’m afraid that I don’t know too much about Kerberos, but it should 
>> be doable with the right custom Authenticator and/or Spawner. The JHU 
>> setup  
>> Carol linked to is probably the place to start. If you have IT folks who 
>> know the moving parts for your Kerberos setup, then we should be able to 
>> work it out. Here are the pieces of information you need:
>>
>>1. How do users authenticate? From the looks of the JHU setup, 
>>PAM authentication will work. 
>>2. How do notebook server processes need to be started? In the 
>>JHU setup, they use sshpass to localhost to start a session for 
>>each user. 
>>3. What information needs to be passed to the notebook server 
>>and/or kernel processes for a given user? Sometimes environment 
>> variables 
>>and/or certificate files need to be loaded. The Spawner can do this, 
>> as 
>>long as you tell it what it needs. 
>>
>> With a bit more details, we should be able to have you up and running.
>>
>> -MinRK
>> ​
>>
>>
>> On Thu, Jun 16, 2016 at 2:31 PM, Gerg  wrote:
>>
>>> Carol-
>>>
>>> I really appreciate that Carol.
>>> Our data scientists are really waiting on getting there favorite 
>>> notebook work with our new kerborised setup. Thanks Carol
>>>
>>> Gerg
>>>
>>>
>>> On Wednesday, June 15, 2016 at 3:14:46 PM UTC-4, Carol Willing wrote:

 Hi Greg,

 Min, the JupyterHub lead dev, is traveling this week. I'm copying 
 him on this message since he may have more insights on 
 JupyterHub/Jupyter 
 with Kerberos. 

 Carol

 ---
 Carol Willing

 Research Software Engineer
 Project Jupyter @ Cal Poly SLO

 Director, Python Software Foundation

 On Jun 15 2016, at 1:22 pm, Gerg  wrote: 

> Hi Carol-
>
> Thanks for your reply. 
> It looks like there is no much information for jupyter to work 
> with kerberos.
>
> Thanks
> Gerg
>
> On Monday, June 13, 2016 at 10:13:05 AM UTC-4, Carol Willing wrote:
>
> Hello Gerg,
>
> I'm hoping that others will jump in with information about using 
> Kerberos.
>
> A thread from earlier this year on Kerberos:
>
> https://groups.google.com/forum/#!searchin/jupyter/kerberos/jupyter/Vfzpx0QstLk/g69c0BpXBwAJ
>  
> 
>
> Within that thread (Apr 27 message from Mark Rosen), there is a 
> configuration that is being used at JHU.
>
> Here's also a message from Min that talks about a custom 
> authenticator that would use Authenticator.pre_spawn_start and 
> Authenticator.post_spawn_stop that should be helpful as well: 
> http://pe

Re: [jupyter] Getting Started with Jupyter on a Kerberized cluster

2016-10-13 Thread Gerg 
Hi Tristan-

I completely agree with you that it is straight forward when you run 
notebook themselves. But with little complications in getting the jupyter 
hub to work in kerborized environment, i would say jupyter hub works 
awesome (Kudos goes to jupyter engineers Min and Carol to name few), after 
the setup it reduces the work on the admin side and new users can easily 
get started. Jupyter hub takes care of all the ticket and negotiations that 
happen while talking.

Cheers,
Gerg

On Thursday, October 13, 2016 at 3:38:47 AM UTC-4, Tristan Zajonc wrote:
>
> Gerg,
>
> If users run the notebook themselves, as they would on their laptop, this 
> should be relatively straightforward.  Users can just run kinit and their 
> keytabs should get picked up by most Hadoop tools like Spark.  In a 
> multi-user or containerized setup it may be more complicated.
>
> Tristan
>
> On Tuesday, October 4, 2016 at 2:41:44 AM UTC-7, Hani1814 wrote:
>>
>> Hi Greg,
>>
>> Is it possible to share the set-up?
>>
>> On Monday, 27 June 2016 12:46:26 UTC+1, Gerg wrote:
>>>
>>> Hi Deepak-
>>>
>>> We are not there yet, but information that min provided will be helpful 
>>> in setting up HUB.
>>> We will update you once we have something
>>>
>>> Thanks
>>>
>>> On Friday, June 24, 2016 at 6:38:25 AM UTC-4, Deepak Subhramanian wrote:

 We are also trying to get Jupyter working with Kerberized cluster. Did 
 anyone managed to get it working ?

 On Thursday, 16 June 2016 05:58:17 UTC-7, Min RK wrote:
>
> I’m afraid that I don’t know too much about Kerberos, but it should be 
> doable with the right custom Authenticator and/or Spawner. The JHU 
> setup  
> Carol linked to is probably the place to start. If you have IT folks who 
> know the moving parts for your Kerberos setup, then we should be able to 
> work it out. Here are the pieces of information you need:
>
>1. How do users authenticate? From the looks of the JHU setup, PAM 
>authentication will work. 
>2. How do notebook server processes need to be started? In the JHU 
>setup, they use sshpass to localhost to start a session for each 
>user. 
>3. What information needs to be passed to the notebook server 
>and/or kernel processes for a given user? Sometimes environment 
> variables 
>and/or certificate files need to be loaded. The Spawner can do this, 
> as 
>long as you tell it what it needs. 
>
> With a bit more details, we should be able to have you up and running.
>
> -MinRK
> ​
>
>
> On Thu, Jun 16, 2016 at 2:31 PM, Gerg  wrote:
>
>> Carol-
>>
>> I really appreciate that Carol.
>> Our data scientists are really waiting on getting there favorite 
>> notebook work with our new kerborised setup. Thanks Carol
>>
>> Gerg
>>
>>
>> On Wednesday, June 15, 2016 at 3:14:46 PM UTC-4, Carol Willing wrote:
>>>
>>> Hi Greg,
>>>
>>> Min, the JupyterHub lead dev, is traveling this week. I'm copying 
>>> him on this message since he may have more insights on 
>>> JupyterHub/Jupyter 
>>> with Kerberos. 
>>>
>>> Carol
>>>
>>> ---
>>> Carol Willing
>>>
>>> Research Software Engineer
>>> Project Jupyter @ Cal Poly SLO
>>>
>>> Director, Python Software Foundation
>>>
>>> On Jun 15 2016, at 1:22 pm, Gerg  wrote: 
>>>
 Hi Carol-

 Thanks for your reply. 
 It looks like there is no much information for jupyter to work with 
 kerberos.

 Thanks
 Gerg

 On Monday, June 13, 2016 at 10:13:05 AM UTC-4, Carol Willing wrote:

 Hello Gerg,

 I'm hoping that others will jump in with information about using 
 Kerberos.

 A thread from earlier this year on Kerberos:

 https://groups.google.com/forum/#!searchin/jupyter/kerberos/jupyter/Vfzpx0QstLk/g69c0BpXBwAJ
  
 

 Within that thread (Apr 27 message from Mark Rosen), there is a 
 configuration that is being used at JHU.

 Here's also a message from Min that talks about a custom 
 authenticator that would use Authenticator.pre_spawn_start and 
 Authenticator.post_spawn_stop that should be helpful as well: 
 http://permalink.gmane.org/gmane.comp.programming.jupyter/2545 
 

 Carol

 ---
 Carol Willing

 Research Software Engineer
 Project Jupyter @ Cal Poly SLO

 Director, Python

Re: [jupyter] Getting Started with Jupyter on a Kerberized cluster

2016-10-13 Thread Tristan Zajonc 
Gerg,

If users run the notebook themselves, as they would on their laptop, this 
should be relatively straightforward.  Users can just run kinit and their 
keytabs should get picked up by most Hadoop tools like Spark.  In a 
multi-user or containerized setup it may be more complicated.

Tristan

On Tuesday, October 4, 2016 at 2:41:44 AM UTC-7, Hani1814 wrote:
>
> Hi Greg,
>
> Is it possible to share the set-up?
>
> On Monday, 27 June 2016 12:46:26 UTC+1, Gerg wrote:
>>
>> Hi Deepak-
>>
>> We are not there yet, but information that min provided will be helpful 
>> in setting up HUB.
>> We will update you once we have something
>>
>> Thanks
>>
>> On Friday, June 24, 2016 at 6:38:25 AM UTC-4, Deepak Subhramanian wrote:
>>>
>>> We are also trying to get Jupyter working with Kerberized cluster. Did 
>>> anyone managed to get it working ?
>>>
>>> On Thursday, 16 June 2016 05:58:17 UTC-7, Min RK wrote:

 I’m afraid that I don’t know too much about Kerberos, but it should be 
 doable with the right custom Authenticator and/or Spawner. The JHU 
 setup  
 Carol linked to is probably the place to start. If you have IT folks who 
 know the moving parts for your Kerberos setup, then we should be able to 
 work it out. Here are the pieces of information you need:

1. How do users authenticate? From the looks of the JHU setup, PAM 
authentication will work. 
2. How do notebook server processes need to be started? In the JHU 
setup, they use sshpass to localhost to start a session for each 
user. 
3. What information needs to be passed to the notebook server 
and/or kernel processes for a given user? Sometimes environment 
 variables 
and/or certificate files need to be loaded. The Spawner can do this, as 
long as you tell it what it needs. 

 With a bit more details, we should be able to have you up and running.

 -MinRK
 ​


 On Thu, Jun 16, 2016 at 2:31 PM, Gerg  wrote:

> Carol-
>
> I really appreciate that Carol.
> Our data scientists are really waiting on getting there favorite 
> notebook work with our new kerborised setup. Thanks Carol
>
> Gerg
>
>
> On Wednesday, June 15, 2016 at 3:14:46 PM UTC-4, Carol Willing wrote:
>>
>> Hi Greg,
>>
>> Min, the JupyterHub lead dev, is traveling this week. I'm copying him 
>> on this message since he may have more insights on JupyterHub/Jupyter 
>> with 
>> Kerberos. 
>>
>> Carol
>>
>> ---
>> Carol Willing
>>
>> Research Software Engineer
>> Project Jupyter @ Cal Poly SLO
>>
>> Director, Python Software Foundation
>>
>> On Jun 15 2016, at 1:22 pm, Gerg  wrote: 
>>
>>> Hi Carol-
>>>
>>> Thanks for your reply. 
>>> It looks like there is no much information for jupyter to work with 
>>> kerberos.
>>>
>>> Thanks
>>> Gerg
>>>
>>> On Monday, June 13, 2016 at 10:13:05 AM UTC-4, Carol Willing wrote:
>>>
>>> Hello Gerg,
>>>
>>> I'm hoping that others will jump in with information about using 
>>> Kerberos.
>>>
>>> A thread from earlier this year on Kerberos:
>>>
>>> https://groups.google.com/forum/#!searchin/jupyter/kerberos/jupyter/Vfzpx0QstLk/g69c0BpXBwAJ
>>>  
>>> 
>>>
>>> Within that thread (Apr 27 message from Mark Rosen), there is a 
>>> configuration that is being used at JHU.
>>>
>>> Here's also a message from Min that talks about a custom 
>>> authenticator that would use Authenticator.pre_spawn_start and 
>>> Authenticator.post_spawn_stop that should be helpful as well: 
>>> http://permalink.gmane.org/gmane.comp.programming.jupyter/2545 
>>> 
>>>
>>> Carol
>>>
>>> ---
>>> Carol Willing
>>>
>>> Research Software Engineer
>>> Project Jupyter @ Cal Poly SLO
>>>
>>> Director, Python Software Foundation
>>>
>>> On Jun 13 2016, at 5:33 am, Gerg  wrote: 
>>>
>>> Hello Guys-
>>>
>>> We are really happy with jupyter for our data scientists. But 
>>> recently we kerberized our cluster (HDP 2.4.0), and we couldn't find 
>>> any 
>>> article or threads in this group on how to setup jupyter for a 
>>> kerberized 
>>> cluster.
>>> Basically we will be running pyspark jobs(yarn mode) from our 
>>> notebook. is this ever possible with Jupyter on a kerberized cluster?
>>> I would really appreciate if someone can help me with the steps in 
>>> configuring jupyter for our kerberized cluster or direct me to any 
>>> article 
>>

Re: [jupyter] Getting Started with Jupyter on a Kerberized cluster

2016-10-04 Thread Hani Abu Rahmeh 
Hi Greg,

Is it possible to share the set-up?

On Monday, 27 June 2016 12:46:26 UTC+1, Gerg wrote:
>
> Hi Deepak-
>
> We are not there yet, but information that min provided will be helpful in 
> setting up HUB.
> We will update you once we have something
>
> Thanks
>
> On Friday, June 24, 2016 at 6:38:25 AM UTC-4, Deepak Subhramanian wrote:
>>
>> We are also trying to get Jupyter working with Kerberized cluster. Did 
>> anyone managed to get it working ?
>>
>> On Thursday, 16 June 2016 05:58:17 UTC-7, Min RK wrote:
>>>
>>> I’m afraid that I don’t know too much about Kerberos, but it should be 
>>> doable with the right custom Authenticator and/or Spawner. The JHU setup 
>>>  Carol 
>>> linked to is probably the place to start. If you have IT folks who know the 
>>> moving parts for your Kerberos setup, then we should be able to work it 
>>> out. Here are the pieces of information you need:
>>>
>>>1. How do users authenticate? From the looks of the JHU setup, PAM 
>>>authentication will work. 
>>>2. How do notebook server processes need to be started? In the JHU 
>>>setup, they use sshpass to localhost to start a session for each 
>>>user. 
>>>3. What information needs to be passed to the notebook server and/or 
>>>kernel processes for a given user? Sometimes environment variables 
>>> and/or 
>>>certificate files need to be loaded. The Spawner can do this, as long as 
>>>you tell it what it needs. 
>>>
>>> With a bit more details, we should be able to have you up and running.
>>>
>>> -MinRK
>>> ​
>>>
>>>
>>> On Thu, Jun 16, 2016 at 2:31 PM, Gerg  wrote:
>>>
 Carol-

 I really appreciate that Carol.
 Our data scientists are really waiting on getting there favorite 
 notebook work with our new kerborised setup. Thanks Carol

 Gerg


 On Wednesday, June 15, 2016 at 3:14:46 PM UTC-4, Carol Willing wrote:
>
> Hi Greg,
>
> Min, the JupyterHub lead dev, is traveling this week. I'm copying him 
> on this message since he may have more insights on JupyterHub/Jupyter 
> with 
> Kerberos. 
>
> Carol
>
> ---
> Carol Willing
>
> Research Software Engineer
> Project Jupyter @ Cal Poly SLO
>
> Director, Python Software Foundation
>
> On Jun 15 2016, at 1:22 pm, Gerg  wrote: 
>
>> Hi Carol-
>>
>> Thanks for your reply. 
>> It looks like there is no much information for jupyter to work with 
>> kerberos.
>>
>> Thanks
>> Gerg
>>
>> On Monday, June 13, 2016 at 10:13:05 AM UTC-4, Carol Willing wrote:
>>
>> Hello Gerg,
>>
>> I'm hoping that others will jump in with information about using 
>> Kerberos.
>>
>> A thread from earlier this year on Kerberos:
>>
>> https://groups.google.com/forum/#!searchin/jupyter/kerberos/jupyter/Vfzpx0QstLk/g69c0BpXBwAJ
>>  
>> 
>>
>> Within that thread (Apr 27 message from Mark Rosen), there is a 
>> configuration that is being used at JHU.
>>
>> Here's also a message from Min that talks about a custom 
>> authenticator that would use Authenticator.pre_spawn_start and 
>> Authenticator.post_spawn_stop that should be helpful as well: 
>> http://permalink.gmane.org/gmane.comp.programming.jupyter/2545 
>> 
>>
>> Carol
>>
>> ---
>> Carol Willing
>>
>> Research Software Engineer
>> Project Jupyter @ Cal Poly SLO
>>
>> Director, Python Software Foundation
>>
>> On Jun 13 2016, at 5:33 am, Gerg  wrote: 
>>
>> Hello Guys-
>>
>> We are really happy with jupyter for our data scientists. But 
>> recently we kerberized our cluster (HDP 2.4.0), and we couldn't find any 
>> article or threads in this group on how to setup jupyter for a 
>> kerberized 
>> cluster.
>> Basically we will be running pyspark jobs(yarn mode) from our 
>> notebook. is this ever possible with Jupyter on a kerberized cluster?
>> I would really appreciate if someone can help me with the steps in 
>> configuring jupyter for our kerberized cluster or direct me to any 
>> article 
>> which describes that.
>>
>> Thanks in advance for any help.
>>
>> Gerg
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to jupyter+u...@googlegroups.com.
>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/j