Hi Shubhang - I have approved this note to go to the dev@ list for Livy as a moderator. I suggest that you subscribe to the list in order to get the responses from the community.
thanks, --larry On Tue, Nov 29, 2022 at 6:09 AM < dev-reject-1669720161.407095.ofnpgmilmjggalahi...@livy.apache.org> wrote: > > To approve: > dev-accept-1669720161.407095.ofnpgmilmjggalahi...@livy.apache.org > To reject: > dev-reject-1669720161.407095.ofnpgmilmjggalahi...@livy.apache.org > To give a reason to reject: > %%% Start comment > %%% End comment > > > > > ---------- Forwarded message ---------- > From: Shubhang Arora <shubhang.ar...@delhivery.com.invalid> > To: dev@livy.apache.org > Cc: > Bcc: > Date: Tue, 29 Nov 2022 16:38:41 +0530 > Subject: [URGENT] Doubts on Spark <> Livy Integration > Hi Livy dev team > > *Context* > I want to enable live interaction with our spark cluster, where users > would be requesting some data over the web/mobile app, and in the backend > we want to run computations over spark and return the result in the API > call, and the response time that I am aiming for is less than 500ms. > > I've some doubts regarding implementing the same using livy > 1. The spark job we have uses the latest spark version (3.3.0) while livy > documentation says it requires at least Spark 1.6 and supports both Scala > 2.10 and 2.11 builds of Spark, while spark 3.3.0 is built with scale 2.12. > Is there any way around this? > > 2. While using the batches api to initiate a spark job, I want to specify > jar files which need to be executed, though I found one open issue (Not > able to submit jars to Livy via batch/session to an already running > session/context. > <https://issues.apache.org/jira/projects/LIVY/issues/LIVY-869?filter=allopenissues>). > Is there any war around this? > > It would be great if you can also share your thoughts on this whole > architecture and would it be possible to guarantee <500ms of response time? > > Thanks & Regards > Shubhang Arora > >
