Re: Maximum limit for akka.frame.size be greater than 500 MB ?

-- > To unsubscribe e-mail: [hidden email] > <http:///user/SendEmail.jtp?type=node=20796=2> > > > > -- > If you reply to this email, your message will be added to the discussion > below

Re: Maximum limit for akka.frame.size be greater than 500 MB ?

Which Spark version are you using? What are you trying to do exactly and what is the input data? As far as I know, akka has been dropped in recent Spark versions. > On 30 Jan 2017, at 00:44, aravasai wrote: > > I have a spark job running on 2 terabytes of data which

Maximum limit for akka.frame.size be greater than 500 MB ?

I have a spark job running on 2 terabytes of data which creates more than 30,000 partitions. As a result, the spark job fails with the error "Map output statuses were 170415722 bytes which exceeds spark.akka.frameSize 52428800 bytes" (For 1 TB data) However, when I increase the akka.frame.size to