I created the JIRA for you (https://issues.apache.org/jira/browse/LIVY-437, but what exactly stopped you from creating a JIRA? We don't want this to be an problem in the future for other users trying to report issues.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
From: Junaid Nasir <jna...@an10.io>
To: user@livy.incubator.apache.org
Date: 01/30/2018 03:44 AM
Subject: Re: Session taking all the available resources even with number
of cores specified
Hello everyone,
I don't have access to create a JIRA issue on livy project. can someone
please create a ticket around this?
Issue: Livy doesn't support "total-executor-cores" settings for spark
standalone cluster
This will enable livy to run multiple spark executors on standalone
cluster.
Thanks,
Junaid
On Mon, Jan 15, 2018 12:22 PM, Junaid Nasir jna...@an10.io wrote:
Hi Alex,
Thank you for taking your time to investigate this. Will create a jira
ticket and try to help with PR.
On Sat, Jan 13, 2018 1:49 AM, Alex Bozarth ajboz...@us.ibm.com wrote:
Hey Junaid,
I did some digging in both Livy and Spark code and you are correct about
the issue being with total-executor-nodes. There is no such conf as
"spark.total.executor.nodes" in Spark as it only exists as a param to
spark-submit (which Spark then maps to the conf "spark.cores.max"). Also
currently Livy does not support that param when constructing it's
spark-submit, which would be useful and I would recommend opening a JIRA
to address it.
I can also point out a few issues with you POST for future reference
while I'm at it:
- The three "livy.spark.*" confs in your POST don't exist and would not
be used, also the value of "executorCores" is mapped to the conf
"spark.executor.cores" making it redundant
- Similarly "numExecutors" and "driverCores" map to
"spark.executor.instances" and "spark.driver.cores" respectively, which
I believe you tried setting with "livy.spark.*" formatted confs
So in summary you should open up a JIRA for Livy to support the
"total-executor-nodes" param for spark-submit and for now you can see if
setting "spark.cores.max" in the conf works as a work-around (though I'm
not hopeful)
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for Junaid Nasir ---01/11/2018 09:54:57 PM---Hi
Alex,It is working correctly when i create spark session Junaid Nasir
---01/11/2018 09:54:57 PM---Hi Alex,It is working correctly when i
create spark session using bin/pyspark. I can create multiple
From: Junaid Nasir <jna...@an10.io>
To: user@livy.incubator.apache.org
Date: 01/11/2018 09:54 PM
Subject: Re: Session taking all the available resources even with number
of cores specified
Hi Alex,
It is working correctly when i create spark session using bin/pyspark. I
can create multiple sessions as well. command which enable this is
"--total-executor-cores 4". I think livy is not passing it to Spark?
We are using DCOS and livy doesn't work with spark mesos deployment, so
had to setup standalone spark cluster.
On Fri, Jan 12, 2018 2:57 AM, Alex Bozarth ajboz...@us.ibm.com wrote:
Hi Junaid,
From my experience this is an issue with the Spark stand-alone
cluster, which is why Livy is recommended to run with YARN
instead, which should allocate resources properly. @Jerry, you
have seen more production uses of LIvy than me, am I correct that
Livy with a stand-alone Spark cluster can't handle multiple
sessions?
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for Junaid Nasir ---01/11/2018 02:22:55
AM---Hi everyone, I am using livy 0.4 with Spark 2.1.0 standalone
Junaid Nasir ---01/11/2018 02:22:55 AM---Hi everyone, I am using
livy 0.4 with Spark 2.1.0 standalone cluster.I can create sessions
From: Junaid Nasir <jna...@an10.io>
To: user@livy.incubator.apache.org
Date: 01/11/2018 02:22 AM
Subject: Session taking all the available resources even with
number of cores specified
Hi everyone,
I am using livy 0.4 with Spark 2.1.0 standalone cluster.
I can create sessions and run jobs. but 1 session takes up all the
available resources. I have tried setting up executorCores, numExecutors
as well as spark.total.executor.cores. this command works fine when
running a session from cmd line
./spark-2.1.0/bin/pyspark --master spark://master:7077 --executor-cores 2
--num-executors 1 --total-executor-cores 4
Not using Mixmax yet?
post request on livy:8998/session
{
"kind": "pyspark",
"proxyUser": "root",
"conf": {
"spark.cassandra.connection.host": "10.128.1.1,10.128.1.2,10.128.1.3",
"spark.executor.cores": 2,
"spark.total.executor.cores": 2,
"livy.spark.driver.cores": 2,
"livy.spark.executor.cores": 2,
"livy.spark.executor.instances": 1
},
"executorMemory": "1G",
"executorCores": 2,
"numExecutors": 1,
"driverCores": 1,
"driverMemory": "1G"
}
Not using Mixmax yet?
Is there any configuration I can do to limit the cores, so that I can run
multiple sessions on same cluster?
Regards
Junaid
