Hi Mobin,
Livy supports Scala 2.11 as well as Spark 2.0+. Can you please elaborate
your problem?
Thanks
Jerry
On Sun, Jul 2, 2017 at 9:19 PM, Mobin Ranjbar
wrote:
> Hi there,
>
>
> I have a problem around using Livy and Apache Spark(Scala 2.10). I have
> Apache Hive that does not support Spark
IIUC, you want to get batch session state even when application was
finished long ago, so you're trying to read the state from recovery log. I
think this is not a proper way, anyway session metadata and session
recovery log will be deleted after timeout. If you want to extend this
expiry date, you
I think you have to make this csv file accessible from Spark cluster,
putting to HDFS is one possible solution.
On Tue, Jul 25, 2017 at 1:26 AM, Joaquín Silva wrote:
> Hello,
>
>
>
> I'm building a BASH program (using Curl) that should run a Spark code
> remotely using Livy. But one of the code
Current Livy doesn't support the things you mentioned here. You can start
multiple Livy in the cluster, but each LivyServer is a standalone service
doesn't aware the existence of others.
On Wed, Jul 26, 2017 at 10:27 AM, Vivek wrote:
> Hi,
>
> We are now considering moving into a uat environment
I'm guessing maybe you hit this issue (
https://issues.apache.org/jira/browse/SPARK-19995). Maybe you can try a
newer version of Spark.
On Fri, Jul 28, 2017 at 3:34 AM, Yeshwanth Jagini
wrote:
> Hi we are running livy on CDH 5.9.2
>
> i am trying to setup Livy 0.3 and running into Kerberos authe
I think you should check Spark application log to see the details, it is
hard for Livy to get actual error from Spark.
On Fri, Aug 11, 2017 at 12:03 PM, Vivek wrote:
> Hi,
>
> Is there anyway to propagate errors from pyspark back to the calling
> program via Livy?
> Currently the Livy logs only
Does it fail constantly? Can you please check if Livy pick the right repl
jar (2.10) for Spark 2.1.1 + Scala 2.1.0?
On Wed, Aug 16, 2017 at 2:45 AM, Alex Rodrigues
wrote:
> Hi Livyers,
>
> I'm hitting a rather nasty error when attempting to initialize a session.
>
> Livy is running on Docker
> L
/*:/apps/livy/conf:/etc/hadoop/conf:
> com.cloudera.livy.server.LivyServer
>
> Please see attached the list of files in /apps/spark-2.1.1_2.10/jars/*,
> and other with Livy jars (present in /apps/livy/jars/*).
>
> The behaviour seems pretty consistent and issue with Hotspot
t; apps/spark-2.1.1_2.10/jars/*:/etc/hadoop/conf -Xmx1g
> org.apache.spark.deploy.SparkSubmit --properties-file /tmp/
> livyConf3184771000880425953.properties --class
> com.cloudera.livy.rsc.driver.RSCDriverBootstrapper spark-internal
>
>
> On Wed, Aug 16, 2017 at 10:33 AM, Saisai Shao
> wrot
Hi all,
We're under progress to make a first Apache release of Livy
(0.4.0-incubating), we really hope you could verify the RC2[1] release
(binary and source) locally and return us the feedbacks.
We will call for an incubation vote next week if everything is fine.
Thanks a lot for your help.
[1
The Apache Livy team is proud to announce Apache Livy version
0.4.0-incubating.
This is the first Livy release after entering the Apache Incubator.
Livy is web service that exposes a REST interface for managing
long running Apache Spark contexts in your cluster. With Livy, new
applications can be
Session recovery should be enabled by configuration. If it is enabled,
session id will be recorded in the persistent storage, and the number will
continue to increment after restart.
On Wed, Sep 6, 2017 at 8:27 PM, Graham Hukill wrote:
> Ah, also very good to know, thanks. That might be a great
If we're using session name, how do we guarantee the uniqueness of this
name?
Thanks
Jerry
On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth wrote:
> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see
I see. So based on this, we should manage a data structure in Livy Server
to keep all the live sessions' name. Also regarding to session recovery, we
should persist this structure to the reliable storage and recover after
restart.
I'm not pretty sure if it is a good feature or not. First because w
Please set "spark.jars.packages" to the package you wanted in batch POST
protocol "conf" field.
Thanks
Jerry
On Tue, Oct 10, 2017 at 3:05 AM, Junaid Nasir wrote:
> more info regarding the problem.
> when i pass {"kind": "pyspark","jars":["datastax:
> spark-cassandra-connector:2.0.1-s_2.11"]} vi
Would you please provide more information about how you create a Livy
session? As for now, Livy only supports spark on yarn and local mode
officially, we don't test on standalone cluster mode, so maybe there's some
issues in it.
On Mon, Oct 16, 2017 at 4:29 AM, Junaid Nasir wrote:
> Hi everyone,
Unfortunately Mesos cluster manager is not supported currently for Livy, we
only tested on local and yarn mode.
On Wed, Oct 25, 2017 at 8:47 PM, Junaid Nasir wrote:
> livy version 0.4
> trying to create interactive session. may be it's not supported with mesos?
>
>
>
> On Wed, Oct 25, 2017 3:55
You can choose to set "livy.spark.master" to "local" and
"livy.spark.deploy-mode" to "client" to start Spark with local mode, in
such case YARN is not required.
Otherwise if you plan to run on YARN, you have to install Hadoop and
configure HADOOP_CONF_DIR in livy-env.sh.
On Thu, Oct 26, 2017 at 9
I think it should be worked, can you please test with 0.4 version of Livy.
Also "conf" should be a map of string key to string value.
"conf" : {"spark.dynamicAllocation.enabled":"false","spark.shuffle.
service.enabled":"false"}
Besides, please be aware in the current Livy we only tested on local
Since you're running with yarn cluster mode, the output from your python
script should be part of your yarn application log. you can get it via yarn
command like yarn log -applicationId , or others like Yarn
UI. Currently there's no Livy api for you to get application log via REST
API.
Thanks
Jerr
rk driver applications but it looks
> like I might have no choice about this.
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* Saisai Shao [mailto:sai.sai.s...@gmail.com]
> *Sent:* 27 November 2017 02:19
> *To:* user@livy.incubator.apache.org
> *Subject:* EXT: Re: How to acce
Livy doesn't add any restriction on how user uses Spark API, so of course
Structured Streaming is supported.
2017-11-29 14:21 GMT+08:00 kant kodali :
> Hi All,
>
> Does Apache Livy support Spark Structured Streaming 2.2.0? If so, any
> examples please? preferably in Java.
>
> Thanks,
> kant
>
> Nice! so if I submit a streaming query over REST can I get the results
> back over REST or Websocket?
>
> On Tue, Nov 28, 2017 at 10:29 PM, Saisai Shao
> wrote:
>
>> Livy doesn't add any restriction on how user uses Spark API, so of course
>> Structured Streaming i
Hi Lauren,
Thanks for the feedbacks. How do you identify this REPL memory issue? If it
is a problem of Scala REPL, then I think we don't have pretty much
solutions for it. Do you have any thought on it?
BTW, which version of Spark are you using?
@Jeff, if it is a problem of Scala REPL, then I be
This feature is targeted for Livy 0.5.0 community version. But we already
back-ported this in HDP 2.6.3, so you can try this feature in HDP 2.6.3.
You can check this doc (
https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md) to
see the API difference for this feature.
2017-12-03
Using this API "public LivyClientBuilder setConf(String key, String
value)" to set Spark configurations you wanted.
2017-12-06 15:34 GMT+08:00 kant kodali :
> Hi All,
>
>
> I do see POST /sessions API where I can pass driverMemory, driverCores,
> executorMemory as part of the request body but I
ory",
> "4g").setConf("driverCores", "4").build();
>
>
> On Wed, Dec 6, 2017 at 12:20 AM, kant kodali wrote:
>
>> Thanks much!
>>
>> On Wed, Dec 6, 2017 at 12:16 AM, Saisai Shao
>> wrote:
>>
>>> Using this API &
I think they are basically the same, you can use either. Why don't you take
a try.
Thanks
Jerry
2018-01-29 12:37 GMT+08:00 Kyunam Kim :
> Basically, which option is equivalent to "spark-submit --jar
> myLib0,jar,myLib1.jar" ?
>
> Thanks,
> Q
>
IIUC, I assume you're using JobAPI rather than REST APIs. Yes, you can
create a session using Java/Scala job API, and then manipulate with Python
APIs.
Thanks
Jerry
2018-02-06 13:43 GMT+08:00 simsicon :
> Hi,
>
> I am pretty new to livy, and I am looking into the livy to do some
> research if li
ontext
>
> <https://livy.incubator.apache.org/docs/latest/api/scala/index.html#org.apache.livy.scalaapi.ScalaJobContext>refers
> to the same concept of session in REST api?
>
> On Tue, Feb 6, 2018 at 3:22 PM, Saisai Shao
> wrote:
>
>> IIUC, I assume you're using JobAPI rather than
This seems like a Scala version issue (as far as I can see from the stack).
Would you please describe the details about how you use Livy and what
configurations you set?
Thanks
Jerry
2018-02-23 6:30 GMT+08:00 David Espinosa :
> Hi all,
>
> I'm a new user in Livy. I've created a scala app that ru
Can you please try 0.5.0 release version (
http://livy.incubator.apache.org/download/). I assume 0.5.0 should have
fixed this concurrent issue.
Thanks
Jerry
2018-02-23 14:42 GMT+08:00 Sudha KS :
> Hi all,
>
> Anybody facing this error?
> I get error running example code (PiJob) compiled on Scala
I think we already supported single node HA for Livy by using incremented
session id? Can you please elaborate your scenario and advantages of using
GUID?
Thanks
Jerry
Alex Bozarth 于2018年5月4日周五 上午4:13写道：
> This sounds similar to a previous email chain from a few months ago. I
> think your answe
Which version of Livy are you using? This bug should be fixed long ago, I
suggest you to try latest released version to see if this issue still
exists.
姚德义 于2018年5月10日周四 下午5:43写道：
> Hi,
> I submit a spark streaming job by Livy, but after one hour Livy kills the
> job. At that time, streaming job
This is by design that Livy doesn't support concurrent execution of
different requests. Because Livy doesn't know if the subsequent requests
are semantically dependent or not.
Rajeshkumar R 于2018年6月3日周日 下午12:08写道：
> Hello
>> I have a livy session which is linked to a spark context. This spark
>>
No, busy means currently there's job running in Spark, so the follow-up
code will wait until the previous job is done.
JF Chen 于2018年6月22日周五 上午11:53写道：
> Can livy execute code when session is in busy state?
>
> Regard,
> Junfeng Chen
>
No, it doesn't support, you should change the code to add a specific Filter
to make it work.
Harsch, Tim 于2018年6月30日周六 上午12:49写道：
> I thought I read somewhere that Livy supports basic auth, but don't see
> configurations for it. Does it?
>
Hi Wandong,
Livy's shared object mechanism mainly used to share objects between
different Livy jobs, this is mainly used for Job API. For example job A
create a object Foo which wants to be accessed by Job B, then user could
store this object Foo into JobContext with a provided name, after that Jo
Probably some network cost should also be counted in. There's no such
configuration for tuning. If you find some performance issue, you can
create a JIRA or even a patch to fix Livy.
Harsch, Tim 于2018年8月1日周三 上午8:04写道：
>
> I have a Livy application that I'm trying to tune as I'm seeing some
> per
If you're hitting the problem of LIVY-424, this is exactly a Scala REPL
issue, not a Livy issue, it is hard to fix in the Livy side.
lk_hadoop 于2018年11月12日周一 上午9:37写道：
> hi,all:
> I meet this issue https://issues.apache.org/jira/browse/LIVY-424
> , anybody know how to resolve it?
> 2018-
I can also help to release a new version. My only concern is that how
mature the thrift module is, shall we enable it by default or leave it
disabled?
Thanks
Saisai
Jeff Zhang 于2019年3月12日周二 上午10:54写道：
> Thanks Marcelo, I can help to test it in zeppelin side which use livy as
> one interpreter.
Like what we did before to support both Scala 2.10 and 2.11 in Livy, I
think we should also have a new module to support 2.12.
于2019年6月4日周二 上午7:40写道：
> Yes, the spark binary we downloaded is built with default Scala 2.12. We
> want to use databricks delta which I think only support Scala 2.12.
ate 2.12 module? Is that just a
> change in POM files or code change is required?
>
> We have release planned for July to upgrade Jupyter and Livy to utilize
> spark 2.4.2. This is blocking us from upgrade.
> From: Saisai Shao mailto:sai.sai.s...@gmail.com>>
> Date: Monday,
migration that was done an year ago and it
> looks like it is mostly Pom changes. If that’s not correct than I might
> need help to perform the upgrade.
>
> Do you have bandwidth to make this change?
>
>
>
>
> From: Saisai Shao mailto:sai.sai.s...@gmail.com&g
Unfortunately there's no such mechanism to inject custom code when session
is started in Livy side. I think you can add some code in Spark side, Spark
has a listener hook "SparkListener", in which there has a hook `
onApplicationStart`, this hook will be called immediately after application
is star
Would you mind listing the steps to reproduce your issue, and how do you
use REST APIs?
Thanks
Saisai
Pavel Sivak 于2019年8月10日周六 上午11:01写道：
> Hi,
> My idea is to create livy session with my library in class path using
> "jars" parameter.
> I'm using REST API to create a session. After sending PO
va for example) - session is
> in *IDLE* status
>
>> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]}
>
>
> I don't understand what can be the difference between my jar and some
> other jars...
>
> On Sun, Aug 11
Hi,
We don't have a test on Windows Linux subsystem, so I'm not exactly sure
the issue is, why don't you open a JIRA ticket about this problem, so that
we could track and fix it.
Thanks
Saisai
César Tenganán 于2019年9月18日周三 上午12:14写道：
> Hi,
>
> We have been working to configure Apache livy-0.5.0
Livy is just a gateway service, I think it is not Livy's responsibility to
provide a sandbox. Also it is super hard to analyze the code from Livy
side. Typically some companies will provide a sand-boxing runtime, like JVM
or others, or put Spark into container for security and isolation.
Thanks
Sa
I don't think current Livy support such behavior, the cluster manager
specified in the conf file is a global configuration which affects all the
created sessions.
Thanks
Saisai
Ravi Shankar 于2020年1月28日周二 上午4:02写道：
> Hey guys,
> Is there a way to start different kind of spark sessions using the
The Apache Livy team is proud to announce the release of Apache Livy
0.7.0-incubating.
Livy is web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. Livy enables
programmatic, fault-tolerant, multi-tenant submission of Spark jobs
from web/mobile
Though we haven't tried such use case before, from my understanding I think
Livy cannot support it, and such usage scenario seems not a typical usage
pattern.
adley 于2020年3月8日周日 下午9:40写道：
> Hi：
> I followed the official example-PiApp.java, and tried to submit and
> run successfully.
> Th
52 matches
Mail list logo
