Hi Nick,
The scope of the PR I submitted is reduced because we can't make sure if it
is really the root cause of the error you faced. You can check out the
discussion on the PR. So I can just change the assert in the code as shown
in the PR.
If you can have a repro, we can go back to see if it
s.com>
Cc: Spark dev list <dev@spark.apache.org>
Date: 2016/12/08 10:56
Subject: Re: Reduce memory usage of UnsafeInMemorySorter
Unfortunately, I don't have a repro, and I'm only seeing this at scale.
But I was able to get around the issue by fiddling with the distribution
il.com>
> Cc:Spark dev list <dev@spark.apache.org>
> Date:2016/12/07 14:27
> Subject:Re: Reduce memory usage of UnsafeInMemorySorter
> --
>
>
>
> This is not supposed to happen. Do you have a repro?
>
>
&g
/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L156
Regards,
Kazuaki Ishizaki
From: Reynold Xin <r...@databricks.com>
To: Nicholas Chammas <nicholas.cham...@gmail.com>
Cc: Spark dev list <dev@spark.apache.org>
Date: 2016/12/07 14:27
Subject: Re: R
This is not supposed to happen. Do you have a repro?
On Tue, Dec 6, 2016 at 6:11 PM, Nicholas Chammas wrote:
> [Re-titling thread.]
>
> OK, I see that the exception from my original email is being triggered
> from this part of UnsafeInMemorySorter:
>
>
[Re-titling thread.]
OK, I see that the exception from my original email is being triggered from
this part of UnsafeInMemorySorter:
https://github.com/apache/spark/blob/v2.0.2/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L209-L212
So I can ask a more