Hi Mich, 1. Each user could create a Livy session (batch or interactive), one session is backed by one Spark application, and the resource quota is the same as normal spark application (configured by spark.executor.cores/memory,. etc), and this will be passed to yarn if running on Yarn. This is basically no different compared to normal spark application. 2. Livy is just a Rest service, inside Livy you could create multiple sessions, each session is mapped to one spark application and resource managed under the cluster manager (like YARN). 3. One user could create one session, this session will be run as the user, it is similar to HiveServer's user impersonation or poxy user.
On Wed, Dec 7, 2016 at 5:51 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Richard. > > I saw your question in the above blog: > > How does Livy proxy the user? Per task? Do you know how quotas are > assigned to users, like how do you stop one Livy user from using all of the > resources available to the Executors? > > My points are: > > > 1. Still don't understand how quotas are assigned to users. Is that > done by YARN? > 2. What will happen if more than one Livy is running on the same > cluster all controlled by the same YARN. how resouces are allocated > > cheers > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 5 December 2016 at 14:50, Richard Startin <richardstar...@outlook.com> > wrote: > >> There is a great write up on Livy at >> http://henning.kropponline.de/2016/11/06/ >> >> On 5 Dec 2016, at 14:34, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Hi, >> >> Has there been any experience using Livy with Spark to share multiple >> Spark contexts? >> >> thanks >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >
