Re: Memory tuning in spark sql

[Ted Yu]

t
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> Thanks
> Arun
>
> --
> *From:* Ted Yu [yuzhih...@gmail.com]
> *Sent:* 04 February 2016 23:37
> *To:* BONGALE, ARUN (Cognizant)
> *Cc:* user
> *Subject:* Re: Memory tuning in spark sql
>
> Can you provide a bit more detail ?
>
> values of the parameters you have tuned
> log snippets from executors
> snippet of your code
>
> Thanks
>
> On Thu, Feb 4, 2016 at 9:48 AM,  <http://redir.aspx?REF=Cvgtpa7SYatX8coIXyW5Vsnc-ZSwAOo_o6sBm3hEmEIBJpaNkC3TCAFtYWlsdG86QVJVTi5CT05HQUxFQGNvZ25pemFudC5jb20.>
> > wrote:
>
>> Hi Sir/madam,
>> Greetings of the day.
>>
>> I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing
>> some issues in reading large(500 mb) file in spark-sql.
>> Sometimes i get heap space error and sometimes the executors fail.
>> i have increased the driver memory, executor memory, kryo serializer
>> buffer size.. etc.. but nothing helps.
>>
>> I Kindly request your help in resolving this issue.
>>
>> Thanks
>> Arun.
>> This e-mail and any files transmitted with it are for the sole use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. If you are not the intended recipient(s), please reply to the
>> sender and destroy all copies of the original message. Any unauthorized
>> review, use, disclosure, dissemination, forwarding, printing or copying of
>> this email, and/or any action taken in reliance on the contents of this
>> e-mail is strictly prohibited and may be unlawful. Where permitted by
>> applicable law, this e-mail and other e-mail communications sent to and
>> from Cognizant e-mail addresses may be monitored.
>>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>

Re: Memory tuning in spark sql

[Ted Yu]

Can you provide a bit more detail ?

values of the parameters you have tuned
log snippets from executors
snippet of your code

Thanks

On Thu, Feb 4, 2016 at 9:48 AM,  wrote:

> Hi Sir/madam,
> Greetings of the day.
>
> I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing
> some issues in reading large(500 mb) file in spark-sql.
> Sometimes i get heap space error and sometimes the executors fail.
> i have increased the driver memory, executor memory, kryo serializer
> buffer size.. etc.. but nothing helps.
>
> I Kindly request your help in resolving this issue.
>
> Thanks
> Arun.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>

Memory tuning in spark sql

[ARUN.BONGALE]

Hi Sir/madam,
Greetings of the day.

I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing some 
issues in reading large(500 mb) file in spark-sql.
Sometimes i get heap space error and sometimes the executors fail.
i have increased the driver memory, executor memory, kryo serializer buffer 
size.. etc.. but nothing helps.

I Kindly request your help in resolving this issue.

Thanks
Arun.
This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.