I started with a download of 1.6.0. These days, we use a self compiled 1.6.2.
On Mon, Jul 4, 2016 at 11:39 AM Ashwin Raaghav <ashraag...@gmail.com> wrote: > I am thinking of any possibilities as to why this could be happening. If > the cores are multi-threaded, should that affect the daemons? Your spark > was built from source code or downloaded as a binary, though that should > not technically change anything? > > On Mon, Jul 4, 2016 at 9:03 PM, Mathieu Longtin <math...@closetwork.org> > wrote: > >> 1.6.1. >> >> I have no idea. SPARK_WORKER_CORES should do the same. >> >> On Mon, Jul 4, 2016 at 11:24 AM Ashwin Raaghav <ashraag...@gmail.com> >> wrote: >> >>> Which version of Spark are you using? 1.6.1? >>> >>> Any ideas as to why it is not working in ours? >>> >>> On Mon, Jul 4, 2016 at 8:51 PM, Mathieu Longtin <math...@closetwork.org> >>> wrote: >>> >>>> 16. >>>> >>>> On Mon, Jul 4, 2016 at 11:16 AM Ashwin Raaghav <ashraag...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I tried what you suggested and started the slave using the following >>>>> command: >>>>> >>>>> start-slave.sh --cores 1 <master> >>>>> >>>>> But it still seems to start as many pyspark daemons as the number of >>>>> cores in the node (1 parent and 3 workers). Limiting it via spark-env.sh >>>>> file by giving SPARK_WORKER_CORES=1 also didn't help. >>>>> >>>>> When you said it helped you and limited it to 2 processes in your >>>>> cluster, how many cores did each machine have? >>>>> >>>>> On Mon, Jul 4, 2016 at 8:22 PM, Mathieu Longtin < >>>>> math...@closetwork.org> wrote: >>>>> >>>>>> It depends on what you want to do: >>>>>> >>>>>> If, on any given server, you don't want Spark to use more than one >>>>>> core, use this to start the workers: SPARK_HOME/sbin/start-slave.sh >>>>>> --cores=1 >>>>>> >>>>>> If you have a bunch of servers dedicated to Spark, but you don't want >>>>>> a driver to use more than one core per server, then: >>>>>> spark.executor.cores=1 >>>>>> tells it not to use more than 1 core per server. However, it seems it >>>>>> will >>>>>> start as many pyspark as there are cores, but maybe not use them. >>>>>> >>>>>> On Mon, Jul 4, 2016 at 10:44 AM Ashwin Raaghav <ashraag...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Mathieu, >>>>>>> >>>>>>> Isn't that the same as setting "spark.executor.cores" to 1? And how >>>>>>> can I specify "--cores=1" from the application? >>>>>>> >>>>>>> On Mon, Jul 4, 2016 at 8:06 PM, Mathieu Longtin < >>>>>>> math...@closetwork.org> wrote: >>>>>>> >>>>>>>> When running the executor, put --cores=1. We use this and I only >>>>>>>> see 2 pyspark process, one seem to be the parent of the other and is >>>>>>>> idle. >>>>>>>> >>>>>>>> In your case, are all pyspark process working? >>>>>>>> >>>>>>>> On Mon, Jul 4, 2016 at 3:15 AM ar7 <ashraag...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am currently using PySpark 1.6.1 in my cluster. When a pyspark >>>>>>>>> application >>>>>>>>> is run, the load on the workers seems to go more than what was >>>>>>>>> given. When I >>>>>>>>> ran top, I noticed that there were too many Pyspark.daemons >>>>>>>>> processes >>>>>>>>> running. There was another mail thread regarding the same: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://mail-archives.apache.org/mod_mbox/spark-user/201606.mbox/%3ccao429hvi3drc-ojemue3x4q1vdzt61htbyeacagtre9yrhs...@mail.gmail.com%3E >>>>>>>>> >>>>>>>>> I followed what was mentioned there, i.e. reduced the number of >>>>>>>>> executor >>>>>>>>> cores and number of executors in one node to 1. But the number of >>>>>>>>> pyspark.daemons process is still not coming down. It looks like >>>>>>>>> initially >>>>>>>>> there is one Pyspark.daemons process and this in turn spawns as >>>>>>>>> many >>>>>>>>> pyspark.daemons processes as the number of cores in the machine. >>>>>>>>> >>>>>>>>> Any help is appreciated :) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ashwin Raaghav. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Limiting-Pyspark-daemons-tp27272.html >>>>>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>>>>> Nabble.com. >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>>>>>> >>>>>>>>> -- >>>>>>>> Mathieu Longtin >>>>>>>> 1-514-803-8977 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Ashwin Raaghav >>>>>>> >>>>>> -- >>>>>> Mathieu Longtin >>>>>> 1-514-803-8977 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Ashwin Raaghav >>>>> >>>> -- >>>> Mathieu Longtin >>>> 1-514-803-8977 >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Ashwin Raaghav >>> >> -- >> Mathieu Longtin >> 1-514-803-8977 >> > > > > -- > Regards, > > Ashwin Raaghav > -- Mathieu Longtin 1-514-803-8977
