No, current Livy interactive session doesn't support running statements concurrently. Because Livy doesn't know if multiple statements have dependencies or not, running concurrently will lead to unexpected results. So submitted statements can only be run one by one.
Thanks Jerry 2017-12-14 14:25 GMT+08:00 Keiji Yoshida <[email protected]>: > Hi Apache Livy developers, > > I would like to ask a question. > > Can Spark jobs which are submitted via Livy run concurrently on a single > Spark application (= Livy session)? > > I set up Spark configuration in accordance with > https://spark.apache.org/docs/2.1.1/job-scheduling.html# > scheduling-within-an-application > so that Spark jobs can run concurrently on a FAIR Spark scheduler and a > FAIR Spark pool. > > However, when I submitted multiple Spark jobs (= Livy statements) by > executing the following command several times, they just ran sequentially > on a FAIR Spark pool. > > [command which I executed several times] > curl -XPOST -H "Content-Type: application/json" > mylivyhost.com:9999/sessions/9999/statements -d '{"code": > "sc.setLocalProperty(\"spark.scheduler.pool\", > \"my-fair-pool\")\nspark.sql(\"select count(1) from mytable\").show()"}' > > It seems for me that a single Livy session submits a Spark job (= Livy > statement) only after the Spark job which has been submitted previously > finished. > > Is my guess true? Can I make Spark jobs which are submitted via Livy run > concurrently in some way? > > I'm using Livy 0.3.0. > > Thanks, > Keiji Yoshida >
