Another way - with this PR https://github.com/apache/incubator-livy/pull/135 Livy will be able to submit multiple statements into the same session and run them concurrently (without manually creating some sort of pool of sessions).
-- Ruslan Dautkhanov On Tue, Oct 15, 2019 at 1:38 PM Coulson, Andy <andrew.coul...@siemens.com> wrote: > We are beginning to do something similar to what you describe and it seems > to work, although it is not very polished. > > > > We basically have a component that acts as a session manager. When a job > needs to run, it queries the /sessions endpoint and looks for any “idle” > sessions. If it finds one, it uses that session id for statement > submission. If it does not, it POSTS to the sessions endpoint to create a > new one (up to a configurable number of total sessions), waits for the new > one to go idle, then returns the sessions id. What we have not implemented > yet is something to go into a wait-loop if the maximum number of sessions > already exist. > > > > *From:* Ravi Shankar <r...@unifisoftware.com> > *Sent:* Tuesday, October 15, 2019 2:29 PM > *To:* user@livy.incubator.apache.org > *Subject:* Long running spark sessions with livy > > > > Hey guys, > > I keep reading that livy supports "Long running spark sessions that can be > used for multiple spark jobs by multiple clients". How do i achieve this > functionality in livy ? I am basically trying to achieve something like > this in our application: > > > > - Have a pre-configured list of spark sessions > > - Have one of them long running for quick preview operations from our > application. i.e , this session should not exit and can be used for > multiple jobs. > > - Have a timeout for the long running session (i.e if no jobs are > submitted within a certain period of time, exit the session automatically) > > > > I see that livy supports a batch session, but doesn't the batch session > exit after the submitted job finishes ? >
