Ok, thanks Reynold. When I tested dynamic allocation with Spark 1.4, it
complained saying that it was not tungsten compliant. Lets hope it works
with 1.5 then!
On Tue, Sep 8, 2015 at 5:49 AM Reynold Xin wrote:
>
> On Wed, Sep 2, 2015 at 12:03 AM, Anders Arpteg wrote:
>
>>
>> BTW, is it possible
On Wed, Sep 2, 2015 at 12:03 AM, Anders Arpteg wrote:
>
> BTW, is it possible (or will it be) to use Tungsten with dynamic
> allocation and the external shuffle manager?
>
>
Yes - I think this already works. There isn't anything specific here
related to Tungsten.
I haven't done a comparative benchmarking between the two, and it would
involve some work to do so. A single run with each suffler would probably
not say that much since we have a rather busy cluster and the performance
heavily depends on what's currently running in the cluster. I have seen
less pr
Thanks for the confirmation. The tungsten-sort is not the default
ShuffleManager, this fix will not block 1.5 release, it may be in
1.5.1.
BTW, How is the difference between sort and tungsten-sort
ShuffleManager for this large job?
On Tue, Sep 1, 2015 at 8:03 AM, Anders Arpteg wrote:
> A fix sub
A fix submitted less than one hour after my mail, very impressive Davies!
I've compiled your PR and tested it with the large job that failed before,
and it seems to work fine now without any exceptions. Awesome, thanks!
Best,
Anders
On Tue, Sep 1, 2015 at 1:38 AM Davies Liu wrote:
> I had sent
I had sent out a PR [1] to fix 2), could you help to test that?
[1] https://github.com/apache/spark/pull/8543
On Mon, Aug 31, 2015 at 12:34 PM, Anders Arpteg wrote:
> Was trying out 1.5 rc2 and noticed some issues with the Tungsten shuffle
> manager. One problem was when using the com.databrick
Was trying out 1.5 rc2 and noticed some issues with the Tungsten shuffle
manager. One problem was when using the com.databricks.spark.avro reader
and the error(1) was received, see stack trace below. The problem does not
occur with the "sort" shuffle manager.
Another problem was in a large complex