Hi Javaguy, I believe what you aim for can be accomplished by uploading your scala code as a jar and calling it in the interactive session. You can either 1. upload the jar upfront to HDFS of any other external service where spark will be able to access it, and refer to it in the "jars" property of the session creation request or 2. use sessions/:id/upload-jar endpoint to add your jar after you started a session.
HTH, Andras On Wed, Nov 4, 2020 at 8:56 AM javaguy Java <[email protected]> wrote: > Hi, > > I'm new to Livy and after a few days of digging through all the examples > and possible combinations I've finally been able to achieve what I want - > namely, I'm trying to create a large DataFrame and then call > createOrReplaceTempView to keep this table in memory that can be re-used > across sessions > > My preferred approach would be to use the http batch api, but after > googling and trialing, the spark session doesn't seem to be capable of > being re-used and most notably I can't send the session id with the batch > job POST across subsequent sessions. Is that correct? > > I also tried to see if it was possible to share sessions via the > programmatic API as well but I didn't see a place where I can set the > session id. > > I have got a shared spark dataframe working via http POST + sessions but > this has its own drawbacks too. I have found that migrating working scala > class code into the json body with string escaping quotes etc. very > cumbersome and very trial and error. If my only choice is to use http > sessions in order to re-use the DataFrame I was wondering if anyone can > advise any best practices on how I might be able to re-use my existing + > compiling scala code into the json code body of my http POST to tighten > that trial and error approach? > > My preferred approach for sharing sessions would be: > 1. batch api via http > 2. programmatic api > 3. sessions via http (only b/c there are no alternatives) > > Would appreciate if someone can advise > > Thx >
