Re: processing 50 gb data using just one machine
50gb of data is not much.
besides master local[4] what else do you have for other parameters?
${SPARK_HOME}/bin/spark-submit \
--driver-memory 4G \
--num-executors 1 \
--executor-memory 4G \
--master local[4] \
Try running it and check web GUI 4040 for resource usage
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 15 June 2016 at 17:03, spR wrote:
> Hi,
>
> can I use spark in local mode using 4 cores to process 50gb data
> effeciently?
>
> Thank you
>
> misha
>
Re: processing 50 gb data using just one machine
Thanks! got that. I was worried about the time itself. On Wed, Jun 15, 2016 at 10:10 AM, Sergio Fernández wrote: > In theory yes... the common sense say that: > > volume / resources = time > > So more volume on the same processing resources would just take more time. > On Jun 15, 2016 6:43 PM, "spR" wrote: > >> I have 16 gb ram, i7 >> >> Will this config be able to handle the processing without my ipythin >> notebook dying? >> >> The local mode is for testing purpose. But, I do not have any cluster at >> my disposal. So can I make this work with the configuration that I have? >> Thank you. >> On Jun 15, 2016 9:40 AM, "Deepak Goel" wrote: >> >>> What do you mean by "EFFECIENTLY"? >>> >>> Hey >>> >>> Namaskara~Nalama~Guten Tag~Bonjour >>> >>> >>>-- >>> Keigu >>> >>> Deepak >>> 73500 12833 >>> www.simtree.net, dee...@simtree.net >>> deic...@gmail.com >>> >>> LinkedIn: www.linkedin.com/in/deicool >>> Skype: thumsupdeicool >>> Google talk: deicool >>> Blog: http://loveandfearless.wordpress.com >>> Facebook: http://www.facebook.com/deicool >>> >>> "Contribute to the world, environment and more : >>> http://www.gridrepublic.org >>> " >>> >>> On Wed, Jun 15, 2016 at 9:33 PM, spR wrote: >>> Hi, can I use spark in local mode using 4 cores to process 50gb data effeciently? Thank you misha >>> >>>
Re: processing 50 gb data using just one machine
I meant local mode is testing purpose generally. But, I have to use the entire 50gb data. On Wed, Jun 15, 2016 at 10:14 AM, Deepak Goel wrote: > If it is just for test purpose, why not use a smaller size of data and > test it on your notebook. When you go for the cluster, you can go for 50GB > (I am a newbie so my thought would be very naive) > > Hey > > Namaskara~Nalama~Guten Tag~Bonjour > > >-- > Keigu > > Deepak > 73500 12833 > www.simtree.net, dee...@simtree.net > deic...@gmail.com > > LinkedIn: www.linkedin.com/in/deicool > Skype: thumsupdeicool > Google talk: deicool > Blog: http://loveandfearless.wordpress.com > Facebook: http://www.facebook.com/deicool > > "Contribute to the world, environment and more : > http://www.gridrepublic.org > " > > On Wed, Jun 15, 2016 at 10:40 PM, Sergio Fernández > wrote: > >> In theory yes... the common sense say that: >> >> volume / resources = time >> >> So more volume on the same processing resources would just take more time. >> On Jun 15, 2016 6:43 PM, "spR" wrote: >> >>> I have 16 gb ram, i7 >>> >>> Will this config be able to handle the processing without my ipythin >>> notebook dying? >>> >>> The local mode is for testing purpose. But, I do not have any cluster at >>> my disposal. So can I make this work with the configuration that I have? >>> Thank you. >>> On Jun 15, 2016 9:40 AM, "Deepak Goel" wrote: >>> What do you mean by "EFFECIENTLY"? Hey Namaskara~Nalama~Guten Tag~Bonjour -- Keigu Deepak 73500 12833 www.simtree.net, dee...@simtree.net deic...@gmail.com LinkedIn: www.linkedin.com/in/deicool Skype: thumsupdeicool Google talk: deicool Blog: http://loveandfearless.wordpress.com Facebook: http://www.facebook.com/deicool "Contribute to the world, environment and more : http://www.gridrepublic.org " On Wed, Jun 15, 2016 at 9:33 PM, spR wrote: > Hi, > > can I use spark in local mode using 4 cores to process 50gb data > effeciently? > > Thank you > > misha > >
Re: processing 50 gb data using just one machine
If it is just for test purpose, why not use a smaller size of data and test it on your notebook. When you go for the cluster, you can go for 50GB (I am a newbie so my thought would be very naive) Hey Namaskara~Nalama~Guten Tag~Bonjour -- Keigu Deepak 73500 12833 www.simtree.net, dee...@simtree.net deic...@gmail.com LinkedIn: www.linkedin.com/in/deicool Skype: thumsupdeicool Google talk: deicool Blog: http://loveandfearless.wordpress.com Facebook: http://www.facebook.com/deicool "Contribute to the world, environment and more : http://www.gridrepublic.org " On Wed, Jun 15, 2016 at 10:40 PM, Sergio Fernández wrote: > In theory yes... the common sense say that: > > volume / resources = time > > So more volume on the same processing resources would just take more time. > On Jun 15, 2016 6:43 PM, "spR" wrote: > >> I have 16 gb ram, i7 >> >> Will this config be able to handle the processing without my ipythin >> notebook dying? >> >> The local mode is for testing purpose. But, I do not have any cluster at >> my disposal. So can I make this work with the configuration that I have? >> Thank you. >> On Jun 15, 2016 9:40 AM, "Deepak Goel" wrote: >> >>> What do you mean by "EFFECIENTLY"? >>> >>> Hey >>> >>> Namaskara~Nalama~Guten Tag~Bonjour >>> >>> >>>-- >>> Keigu >>> >>> Deepak >>> 73500 12833 >>> www.simtree.net, dee...@simtree.net >>> deic...@gmail.com >>> >>> LinkedIn: www.linkedin.com/in/deicool >>> Skype: thumsupdeicool >>> Google talk: deicool >>> Blog: http://loveandfearless.wordpress.com >>> Facebook: http://www.facebook.com/deicool >>> >>> "Contribute to the world, environment and more : >>> http://www.gridrepublic.org >>> " >>> >>> On Wed, Jun 15, 2016 at 9:33 PM, spR wrote: >>> Hi, can I use spark in local mode using 4 cores to process 50gb data effeciently? Thank you misha >>> >>>
Re: processing 50 gb data using just one machine
In theory yes... the common sense say that: volume / resources = time So more volume on the same processing resources would just take more time. On Jun 15, 2016 6:43 PM, "spR" wrote: > I have 16 gb ram, i7 > > Will this config be able to handle the processing without my ipythin > notebook dying? > > The local mode is for testing purpose. But, I do not have any cluster at > my disposal. So can I make this work with the configuration that I have? > Thank you. > On Jun 15, 2016 9:40 AM, "Deepak Goel" wrote: > >> What do you mean by "EFFECIENTLY"? >> >> Hey >> >> Namaskara~Nalama~Guten Tag~Bonjour >> >> >>-- >> Keigu >> >> Deepak >> 73500 12833 >> www.simtree.net, dee...@simtree.net >> deic...@gmail.com >> >> LinkedIn: www.linkedin.com/in/deicool >> Skype: thumsupdeicool >> Google talk: deicool >> Blog: http://loveandfearless.wordpress.com >> Facebook: http://www.facebook.com/deicool >> >> "Contribute to the world, environment and more : >> http://www.gridrepublic.org >> " >> >> On Wed, Jun 15, 2016 at 9:33 PM, spR wrote: >> >>> Hi, >>> >>> can I use spark in local mode using 4 cores to process 50gb data >>> effeciently? >>> >>> Thank you >>> >>> misha >>> >> >>
Re: processing 50 gb data using just one machine
I have 16 gb ram, i7 Will this config be able to handle the processing without my ipythin notebook dying? The local mode is for testing purpose. But, I do not have any cluster at my disposal. So can I make this work with the configuration that I have? Thank you. On Jun 15, 2016 9:40 AM, "Deepak Goel" wrote: > What do you mean by "EFFECIENTLY"? > > Hey > > Namaskara~Nalama~Guten Tag~Bonjour > > >-- > Keigu > > Deepak > 73500 12833 > www.simtree.net, dee...@simtree.net > deic...@gmail.com > > LinkedIn: www.linkedin.com/in/deicool > Skype: thumsupdeicool > Google talk: deicool > Blog: http://loveandfearless.wordpress.com > Facebook: http://www.facebook.com/deicool > > "Contribute to the world, environment and more : > http://www.gridrepublic.org > " > > On Wed, Jun 15, 2016 at 9:33 PM, spR wrote: > >> Hi, >> >> can I use spark in local mode using 4 cores to process 50gb data >> effeciently? >> >> Thank you >> >> misha >> > >
processing 50 gb data using just one machine
Hi, can I use spark in local mode using 4 cores to process 50gb data effeciently? Thank you misha
