Re: Reduce memory usage of UnsafeInMemorySorter

Hi Nick, The scope of the PR I submitted is reduced because we can't make sure if it is really the root cause of the error you faced. You can check out the discussion on the PR. So I can just change the assert in the code as shown in the PR. If you can have a repro, we can go back to see if it

Re: Reduce memory usage of UnsafeInMemorySorter

s.com> Cc: Spark dev list <dev@spark.apache.org> Date: 2016/12/08 10:56 Subject: Re: Reduce memory usage of UnsafeInMemorySorter Unfortunately, I don't have a repro, and I'm only seeing this at scale. But I was able to get around the issue by fiddling with the distribution

Re: Reduce memory usage of UnsafeInMemorySorter

il.com> > Cc:Spark dev list <dev@spark.apache.org> > Date:2016/12/07 14:27 > Subject:Re: Reduce memory usage of UnsafeInMemorySorter > -- > > > > This is not supposed to happen. Do you have a repro? > > &g

Re: Reduce memory usage of UnsafeInMemorySorter

/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L156 Regards, Kazuaki Ishizaki From: Reynold Xin <r...@databricks.com> To: Nicholas Chammas <nicholas.cham...@gmail.com> Cc: Spark dev list <dev@spark.apache.org> Date: 2016/12/07 14:27 Subject: Re: R

Re: Reduce memory usage of UnsafeInMemorySorter

This is not supposed to happen. Do you have a repro? On Tue, Dec 6, 2016 at 6:11 PM, Nicholas Chammas wrote: > [Re-titling thread.] > > OK, I see that the exception from my original email is being triggered > from this part of UnsafeInMemorySorter: > >

Reduce memory usage of UnsafeInMemorySorter

[Re-titling thread.] OK, I see that the exception from my original email is being triggered from this part of UnsafeInMemorySorter: https://github.com/apache/spark/blob/v2.0.2/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L209-L212 So I can ask a more