Hi everyone,
I am using livy 0.4 with Spark 2.1.0 standalone cluster.I can create sessions
and run jobs. but 1 session takes up all the available resources. I have tried
setting up executorCores, numExecutors as well as spark.total.executor.cores.
this command works fine when running a session from cmd line
./spark-2.1.0/bin/pyspark --master spark://master:7077
--executor-cores 2 --num-executors 1 --total-executor-cores 4
Not using Mixmax yet?
post request on livy:8998/session
{ "kind": "pyspark", "proxyUser": "root", "conf": {
"spark.cassandra.connection.host": "10.128.1.1,10.128.1.2,10.128.1.3",
"spark.executor.cores": 2, "spark.total.executor.cores": 2,
"livy.spark.driver.cores": 2, "livy.spark.executor.cores": 2,
"livy.spark.executor.instances": 1 }, "executorMemory": "1G",
"executorCores": 2, "numExecutors": 1, "driverCores": 1,
"driverMemory": "1G"}
Not using Mixmax yet?
Is there any configuration I can do to limit the cores, so that I can run
multiple sessions on same cluster?
RegardsJunaid
