Re: A new link request to my project and one question
I think Livy super user is similar to Hadoop's proxy user, it allows this user to impersonate others, but it doesn't check whether other users is allowed to be impersonated. In the meantime, Livy has ACL mechanisms, which allows only ACL verified users to connect to LivyServer, so I think with ACL, we can do a more fine-grained control. For other missing point, I think we can improve the Livy code. Marcelo Vanzin 于2018年6月26日周二 上午9:53写道： > Superusers are a little more than "allowed to impersonate others". I > don't remember exactly what are the things that it allows, but it > would be better to add finer grained permissions. > > On Mon, Jun 25, 2018 at 6:30 PM, Saisai Shao > wrote: > > Yes, has a configuration "livy.superusers". Here in this case, the sql > > server user should be added as a superuser, who can impersonate other > > different users. > > > > Marcelo Vanzin 于2018年6月26日周二 上午9:12写道： > > > >> You're talking about another service between the user and the > application. > >> > >> In that case a parameter probably makes sense. But then you'd need to > >> add those config options, because this is a dangerous feature, and > >> Livy should know who is allowed to impersonate who. In this case the > >> service needs to authenticate to Livy as a privileged user, and Livy's > >> configuration would say that the service's user is allowed to > >> impersonate certain users or groups (same as the other services that > >> allow impersonation like YARN). > >> > >> > >> On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro < > linguin@gmail.com> > >> wrote: > >> > Yea, I know the Livy supports impersonation. > >> > I assume a case blow > >> > [different users] ---Some protocols---> [the server applications > managing > >> > multiple sessions for users] ---REST---> [Livy server] > >> > In this case, Livy already has a way to pass proxyUser from the > >> application > >> > to Livy? > >> > Sorry, but I'm not familiar with Livy internal logic. > >> > > >> > > >> > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > >> > >> > wrote: > >> > > >> >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro < > >> linguin@gmail.com> > >> >> wrote: > >> >> > In that case, I think Livy is useful; the application can pass > >> proxyUser > >> >> to > >> >> > build LivyClient for each user > >> >> > and run spark queries as each user authorization. > >> >> > >> >> But Livy already supports impersonation. It can impersonate the > >> >> authenticated user. > >> >> > >> >> You're suggesting adding a parameter so the user can request > >> >> impersonation of some specific user, which is a different thing. What > >> >> is the use case for that? > >> >> > >> >> -- > >> >> Marcelo > >> >> > >> > > >> > > >> > -- > >> > --- > >> > Takeshi Yamamuro > >> > >> > >> > >> -- > >> Marcelo > >> > > > > -- > Marcelo >
Re: A new link request to my project and one question
Superusers are a little more than "allowed to impersonate others". I don't remember exactly what are the things that it allows, but it would be better to add finer grained permissions. On Mon, Jun 25, 2018 at 6:30 PM, Saisai Shao wrote: > Yes, has a configuration "livy.superusers". Here in this case, the sql > server user should be added as a superuser, who can impersonate other > different users. > > Marcelo Vanzin 于2018年6月26日周二 上午9:12写道： > >> You're talking about another service between the user and the application. >> >> In that case a parameter probably makes sense. But then you'd need to >> add those config options, because this is a dangerous feature, and >> Livy should know who is allowed to impersonate who. In this case the >> service needs to authenticate to Livy as a privileged user, and Livy's >> configuration would say that the service's user is allowed to >> impersonate certain users or groups (same as the other services that >> allow impersonation like YARN). >> >> >> On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro >> wrote: >> > Yea, I know the Livy supports impersonation. >> > I assume a case blow >> > [different users] ---Some protocols---> [the server applications managing >> > multiple sessions for users] ---REST---> [Livy server] >> > In this case, Livy already has a way to pass proxyUser from the >> application >> > to Livy? >> > Sorry, but I'm not familiar with Livy internal logic. >> > >> > >> > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin >> >> > wrote: >> > >> >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro < >> linguin@gmail.com> >> >> wrote: >> >> > In that case, I think Livy is useful; the application can pass >> proxyUser >> >> to >> >> > build LivyClient for each user >> >> > and run spark queries as each user authorization. >> >> >> >> But Livy already supports impersonation. It can impersonate the >> >> authenticated user. >> >> >> >> You're suggesting adding a parameter so the user can request >> >> impersonation of some specific user, which is a different thing. What >> >> is the use case for that? >> >> >> >> -- >> >> Marcelo >> >> >> > >> > >> > -- >> > --- >> > Takeshi Yamamuro >> >> >> >> -- >> Marcelo >> -- Marcelo
Re: A new link request to my project and one question
oh, great news... I'll check the config. On Tue, Jun 26, 2018 at 10:30 AM Saisai Shao wrote: > Yes, has a configuration "livy.superusers". Here in this case, the sql > server user should be added as a superuser, who can impersonate other > different users. > > Marcelo Vanzin 于2018年6月26日周二 上午9:12写道： > > > You're talking about another service between the user and the > application. > > > > In that case a parameter probably makes sense. But then you'd need to > > add those config options, because this is a dangerous feature, and > > Livy should know who is allowed to impersonate who. In this case the > > service needs to authenticate to Livy as a privileged user, and Livy's > > configuration would say that the service's user is allowed to > > impersonate certain users or groups (same as the other services that > > allow impersonation like YARN). > > > > > > On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro > > > wrote: > > > Yea, I know the Livy supports impersonation. > > > I assume a case blow > > > [different users] ---Some protocols---> [the server applications > managing > > > multiple sessions for users] ---REST---> [Livy server] > > > In this case, Livy already has a way to pass proxyUser from the > > application > > > to Livy? > > > Sorry, but I'm not familiar with Livy internal logic. > > > > > > > > > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > > > > > wrote: > > > > > >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro < > > linguin@gmail.com> > > >> wrote: > > >> > In that case, I think Livy is useful; the application can pass > > proxyUser > > >> to > > >> > build LivyClient for each user > > >> > and run spark queries as each user authorization. > > >> > > >> But Livy already supports impersonation. It can impersonate the > > >> authenticated user. > > >> > > >> You're suggesting adding a parameter so the user can request > > >> impersonation of some specific user, which is a different thing. What > > >> is the use case for that? > > >> > > >> -- > > >> Marcelo > > >> > > > > > > > > > -- > > > --- > > > Takeshi Yamamuro > > > > > > > > -- > > Marcelo > > > -- --- Takeshi Yamamuro
Re: A new link request to my project and one question
Yes, has a configuration "livy.superusers". Here in this case, the sql server user should be added as a superuser, who can impersonate other different users. Marcelo Vanzin 于2018年6月26日周二 上午9:12写道： > You're talking about another service between the user and the application. > > In that case a parameter probably makes sense. But then you'd need to > add those config options, because this is a dangerous feature, and > Livy should know who is allowed to impersonate who. In this case the > service needs to authenticate to Livy as a privileged user, and Livy's > configuration would say that the service's user is allowed to > impersonate certain users or groups (same as the other services that > allow impersonation like YARN). > > > On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro > wrote: > > Yea, I know the Livy supports impersonation. > > I assume a case blow > > [different users] ---Some protocols---> [the server applications managing > > multiple sessions for users] ---REST---> [Livy server] > > In this case, Livy already has a way to pass proxyUser from the > application > > to Livy? > > Sorry, but I'm not familiar with Livy internal logic. > > > > > > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > > > wrote: > > > >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro < > linguin@gmail.com> > >> wrote: > >> > In that case, I think Livy is useful; the application can pass > proxyUser > >> to > >> > build LivyClient for each user > >> > and run spark queries as each user authorization. > >> > >> But Livy already supports impersonation. It can impersonate the > >> authenticated user. > >> > >> You're suggesting adding a parameter so the user can request > >> impersonation of some specific user, which is a different thing. What > >> is the use case for that? > >> > >> -- > >> Marcelo > >> > > > > > > -- > > --- > > Takeshi Yamamuro > > > > -- > Marcelo >
Re: A new link request to my project and one question
> You're talking about another service between the user and the application. yes, I pointed out the case. > In that case a parameter probably makes sense. But then you'd need to > add those config options, yea, I see. Currently, any approach to avoid the dangerous case for the service? I think we can't control who is allowed to impersonate who in the Livy side. Or , In Livy, the use-case I pointed out is out-of-scope? On Tue, Jun 26, 2018 at 10:12 AM Marcelo Vanzin wrote: > You're talking about another service between the user and the application. > > In that case a parameter probably makes sense. But then you'd need to > add those config options, because this is a dangerous feature, and > Livy should know who is allowed to impersonate who. In this case the > service needs to authenticate to Livy as a privileged user, and Livy's > configuration would say that the service's user is allowed to > impersonate certain users or groups (same as the other services that > allow impersonation like YARN). > > > On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro > wrote: > > Yea, I know the Livy supports impersonation. > > I assume a case blow > > [different users] ---Some protocols---> [the server applications managing > > multiple sessions for users] ---REST---> [Livy server] > > In this case, Livy already has a way to pass proxyUser from the > application > > to Livy? > > Sorry, but I'm not familiar with Livy internal logic. > > > > > > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > > > wrote: > > > >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro < > linguin@gmail.com> > >> wrote: > >> > In that case, I think Livy is useful; the application can pass > proxyUser > >> to > >> > build LivyClient for each user > >> > and run spark queries as each user authorization. > >> > >> But Livy already supports impersonation. It can impersonate the > >> authenticated user. > >> > >> You're suggesting adding a parameter so the user can request > >> impersonation of some specific user, which is a different thing. What > >> is the use case for that? > >> > >> -- > >> Marcelo > >> > > > > > > -- > > --- > > Takeshi Yamamuro > > > > -- > Marcelo > -- --- Takeshi Yamamuro
Re: A new link request to my project and one question
You're talking about another service between the user and the application. In that case a parameter probably makes sense. But then you'd need to add those config options, because this is a dangerous feature, and Livy should know who is allowed to impersonate who. In this case the service needs to authenticate to Livy as a privileged user, and Livy's configuration would say that the service's user is allowed to impersonate certain users or groups (same as the other services that allow impersonation like YARN). On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro wrote: > Yea, I know the Livy supports impersonation. > I assume a case blow > [different users] ---Some protocols---> [the server applications managing > multiple sessions for users] ---REST---> [Livy server] > In this case, Livy already has a way to pass proxyUser from the application > to Livy? > Sorry, but I'm not familiar with Livy internal logic. > > > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > wrote: > >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro >> wrote: >> > In that case, I think Livy is useful; the application can pass proxyUser >> to >> > build LivyClient for each user >> > and run spark queries as each user authorization. >> >> But Livy already supports impersonation. It can impersonate the >> authenticated user. >> >> You're suggesting adding a parameter so the user can request >> impersonation of some specific user, which is a different thing. What >> is the use case for that? >> >> -- >> Marcelo >> > > > -- > --- > Takeshi Yamamuro -- Marcelo
Re: A new link request to my project and one question
Yea, I know the Livy supports impersonation. I assume a case blow [different users] ---Some protocols---> [the server applications managing multiple sessions for users] ---REST---> [Livy server] In this case, Livy already has a way to pass proxyUser from the application to Livy? Sorry, but I'm not familiar with Livy internal logic. On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin wrote: > On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro > wrote: > > In that case, I think Livy is useful; the application can pass proxyUser > to > > build LivyClient for each user > > and run spark queries as each user authorization. > > But Livy already supports impersonation. It can impersonate the > authenticated user. > > You're suggesting adding a parameter so the user can request > impersonation of some specific user, which is a different thing. What > is the use case for that? > > -- > Marcelo > -- --- Takeshi Yamamuro
Re: A new link request to my project and one question
On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro wrote: > In that case, I think Livy is useful; the application can pass proxyUser to > build LivyClient for each user > and run spark queries as each user authorization. But Livy already supports impersonation. It can impersonate the authenticated user. You're suggesting adding a parameter so the user can request impersonation of some specific user, which is a different thing. What is the use case for that? -- Marcelo
Re: A new link request to my project and one question
>> Marcelo Sorry, I missed your response. Yea, thanks for your suggestion. >> Meisam I assume one application handles requests from different users, and the request in the application runs spark queries as each user authorization. Since SparkContext currently doesn't support impersionation, I think one solution is that the application manages multiples SparkContext for that. For example, spark thrift-server doesn't support impersonation because of the same reason above: e.g., SPARK-5159, SPARK-11248, and SPARK-21918 In that case, I think Livy is useful; the application can pass proxyUser to build LivyClient for each user and run spark queries as each user authorization. On Tue, Jun 26, 2018 at 5:33 AM Meisam Fathi wrote: > What is the use case for passing the proxy user to LivyClientBuilder? > > On Fri, Jun 15, 2018 at 9:02 AM Marcelo Vanzin > > wrote: > > > re: proxy user, you have to be extremely careful with that. > > > > Livy currently supports proxy user, but for the server only. It allows > > the server to impersonate anyone, so that sessions can run as the > > requesting user. > > > > If you let the user decide who the session will be run as, you'll need > > to add configuration, just as those available in HDFS, YARN, etc, to > > tell Livy which users can impersonate which other users. Otherwise > > you're basically making authentication meaningless. > > > > > > On Thu, Jun 14, 2018 at 7:36 PM, Saisai Shao > > wrote: > > > Sure, I will merge the website code, thanks! > > > > > > For proxyUser thing, I think there's no particular reason not adding > it, > > > maybe we just forgot to add the proxyUser support. > > > > > > It would be better if you could create a JIRA to track this issue. If > > > you're familiar with Livy code, you can also submit a PR about it. > > > > > > Thanks > > > Jerry > > > > > > Takeshi Yamamuro 于2018年6月15日周五 上午7:33写道： > > > > > >> Hi, Livy dev, > > >> > > >> I opened a new pr in incubator-livy-website to add a new link in > > >> third-party-projects.md. It'd be great if you could check this; > > >> https://github.com/apache/incubator-livy-website/pull/23 > > >> > > >> Btw, I have one question; currently, we cannot pass proxyUser > > >> in LivyClientBuilder. Any reason not to add code for that? > > >> I know we can handle this in an application side by adding a bit code > > like > > >> > > >> > > > https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java > > >> But, If Livy itself supported this, it'd be nice to me. > > >> > > >> Best, > > >> takeshi > > >> > > >> -- > > >> --- > > >> Takeshi Yamamuro > > >> > > > > > > > > -- > > Marcelo > > > -- --- Takeshi Yamamuro
Re: A new link request to my project and one question
What is the use case for passing the proxy user to LivyClientBuilder? On Fri, Jun 15, 2018 at 9:02 AM Marcelo Vanzin wrote: > re: proxy user, you have to be extremely careful with that. > > Livy currently supports proxy user, but for the server only. It allows > the server to impersonate anyone, so that sessions can run as the > requesting user. > > If you let the user decide who the session will be run as, you'll need > to add configuration, just as those available in HDFS, YARN, etc, to > tell Livy which users can impersonate which other users. Otherwise > you're basically making authentication meaningless. > > > On Thu, Jun 14, 2018 at 7:36 PM, Saisai Shao > wrote: > > Sure, I will merge the website code, thanks! > > > > For proxyUser thing, I think there's no particular reason not adding it, > > maybe we just forgot to add the proxyUser support. > > > > It would be better if you could create a JIRA to track this issue. If > > you're familiar with Livy code, you can also submit a PR about it. > > > > Thanks > > Jerry > > > > Takeshi Yamamuro 于2018年6月15日周五 上午7:33写道： > > > >> Hi, Livy dev, > >> > >> I opened a new pr in incubator-livy-website to add a new link in > >> third-party-projects.md. It'd be great if you could check this; > >> https://github.com/apache/incubator-livy-website/pull/23 > >> > >> Btw, I have one question; currently, we cannot pass proxyUser > >> in LivyClientBuilder. Any reason not to add code for that? > >> I know we can handle this in an application side by adding a bit code > like > >> > >> > https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java > >> But, If Livy itself supported this, it'd be nice to me. > >> > >> Best, > >> takeshi > >> > >> -- > >> --- > >> Takeshi Yamamuro > >> > > > > -- > Marcelo >
Re: A new link request to my project and one question
re: proxy user, you have to be extremely careful with that. Livy currently supports proxy user, but for the server only. It allows the server to impersonate anyone, so that sessions can run as the requesting user. If you let the user decide who the session will be run as, you'll need to add configuration, just as those available in HDFS, YARN, etc, to tell Livy which users can impersonate which other users. Otherwise you're basically making authentication meaningless. On Thu, Jun 14, 2018 at 7:36 PM, Saisai Shao wrote: > Sure, I will merge the website code, thanks! > > For proxyUser thing, I think there's no particular reason not adding it, > maybe we just forgot to add the proxyUser support. > > It would be better if you could create a JIRA to track this issue. If > you're familiar with Livy code, you can also submit a PR about it. > > Thanks > Jerry > > Takeshi Yamamuro 于2018年6月15日周五 上午7:33写道： > >> Hi, Livy dev, >> >> I opened a new pr in incubator-livy-website to add a new link in >> third-party-projects.md. It'd be great if you could check this; >> https://github.com/apache/incubator-livy-website/pull/23 >> >> Btw, I have one question; currently, we cannot pass proxyUser >> in LivyClientBuilder. Any reason not to add code for that? >> I know we can handle this in an application side by adding a bit code like >> >> https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java >> But, If Livy itself supported this, it'd be nice to me. >> >> Best, >> takeshi >> >> -- >> --- >> Takeshi Yamamuro >> -- Marcelo
Re: A new link request to my project and one question
Thanks for quick reply! > It would be better if you could create a JIRA to track this issue. If > you're familiar with Livy code, you can also submit a PR about it. Aha, ok. ok, I'll file jira and try to make a pr later. Thanks! On Fri, Jun 15, 2018 at 11:36 AM, Saisai Shao wrote: > Sure, I will merge the website code, thanks! > > For proxyUser thing, I think there's no particular reason not adding it, > maybe we just forgot to add the proxyUser support. > > It would be better if you could create a JIRA to track this issue. If > you're familiar with Livy code, you can also submit a PR about it. > > Thanks > Jerry > > Takeshi Yamamuro 于2018年6月15日周五 上午7:33写道： > > > Hi, Livy dev, > > > > I opened a new pr in incubator-livy-website to add a new link in > > third-party-projects.md. It'd be great if you could check this; > > https://github.com/apache/incubator-livy-website/pull/23 > > > > Btw, I have one question; currently, we cannot pass proxyUser > > in LivyClientBuilder. Any reason not to add code for that? > > I know we can handle this in an application side by adding a bit code > like > > > > https://github.com/maropu/spark-sql-server/blob/master/ > sql/sql-server/src/main/java/org/apache/livyclient/common/ > CreateClientRequestWithProxyUser.java > > But, If Livy itself supported this, it'd be nice to me. > > > > Best, > > takeshi > > > > -- > > --- > > Takeshi Yamamuro > > > -- --- Takeshi Yamamuro
Re: A new link request to my project and one question
Sure, I will merge the website code, thanks! For proxyUser thing, I think there's no particular reason not adding it, maybe we just forgot to add the proxyUser support. It would be better if you could create a JIRA to track this issue. If you're familiar with Livy code, you can also submit a PR about it. Thanks Jerry Takeshi Yamamuro 于2018年6月15日周五 上午7:33写道： > Hi, Livy dev, > > I opened a new pr in incubator-livy-website to add a new link in > third-party-projects.md. It'd be great if you could check this; > https://github.com/apache/incubator-livy-website/pull/23 > > Btw, I have one question; currently, we cannot pass proxyUser > in LivyClientBuilder. Any reason not to add code for that? > I know we can handle this in an application side by adding a bit code like > > https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java > But, If Livy itself supported this, it'd be nice to me. > > Best, > takeshi > > -- > --- > Takeshi Yamamuro >
