Re: Use existing SparkSession in POST/batches request

Interactive sessions also allow you to upload jars. Consider trying to transform your batch job, or wrapping your job, into interactive code by giving your batch job an entry point method. class JumboBatchRunner { public static runBatch( SparkContext sc, String startDate, String endDate,

Re: about LIVY-424

While it's true LIVY-424 creates a session leak due to REPL leak in Scala it's not the only thing that can. I've run hundreds of simple scala commands and the leak is only mild/moderate. However, some scala commands can be really problematic. For instance import org.apache.spark.sql._

Re: How to tune Livy for fast queries

. From: Harsch, Tim Sent: Thursday, August 2, 2018 8:28:58 AM To: user@livy.incubator.apache.org Subject: Re: How to tune Livy for fast queries Thank you Saisai for your response. I did have a chance to investigate further and I should give a little

Re: How to tune Livy for fast queries

__ Probably some network cost should also be counted in. There's no such configuration for tuning. If you find some performance issue, you can create a JIRA or even a patch to fix Livy. Harsch, Tim mailto:tim.har...@teradata.com>> 于2018年8月1日周三 上午8:04写道： I have a Livy application th

How to tune Livy for fast queries

I have a Livy application that I'm trying to tune as I'm seeing some performance issue when the queries are fast queries. I've wrapped my queries with a timer that logs the time taken. The spark code executed typically takes 50ms to 150ms. I'm querying Livy every 500ms looking for my

Re: Query: Local Path cannot read by Livy (Running with Docker)

cannot read by Livy (Running with Docker) [External Email] Yes, that is the exact same scenario I have, just instead of jar files I am using CSV files stored in my client machine's file system and no protocol is supporting to access those files. On Sat, Jul 28, 2018 at 3:42 AM, Harsch, Tim

Re: Query: Local Path cannot read by Livy (Running with Docker)

When you say not able to use my files in Spark Session what do you mean? I think I’m seeing a similar problem. If I submit jars using a client machine that is not the same as the server (so file system is not shared) and I submit the jar it’s looks like it gets transmitted to the server but

Re: user impersonation in Livy not working

in livy.conf Harsch, Tim mailto:tim.har...@teradata.com>>于2018年7月10日周二 下午11:53写道： Hi, I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server log will usual

user impersonation in Livy not working

Hi, I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server log will usually say something like: “owner: kylo, proxyUser: Some(dladmin)” so it seems the server

Does Livy Support Basic Auth?

I thought I read somewhere that Livy supports basic auth, but don't see configurations for it. Does it?

Re: Query regarding livy session

If you can guarantee that the statements you are executing are semantically independent, you could create a pool of sessions and execute on any idle session. Not sure in terms of resources how efficient that would be, depends on the cluster configuration in part. From: Saisai Shao Reply-To:

Re: Spark scala examples

Hi Nirav, There are a few ways to use Livy: Shared sessions and Batch sessions are covered in the REST API docs (https://livy.incubator.apache.org/docs/latest/rest-api.html) and GetHue.com wrote some cool articles. I created a little bash framework for testing out some example code I found at

Re: new user to livy/spark, basic use case questions

Results for any given script, or script segments, sent to Livy are retrievable in a subsequent call to the statements endpoint. The results are available as long as the session and server are still alive. If you want a script to write results to a datasource, so they are more permanent, you

%json magick of dataframe drops the datatype information

Hi, I'm trying to understand why, when I use %json magic, dataType is empty? As you can see in this gist: https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72 all structs in the schema come back with just

completion endpoint

Hi, I noticed this endpoint in the REST API documentation. POST /sessions/{sessionId}/completion I've experimented with it a bit, and having a little trouble getting the results to match my expectations. Can someone provide some clues on where I might find example code, or is there some