OK, thank you. What do you suggest I do to get rid of the error?
From: Ted Yu <yuzhih...@gmail.com>
Sent: Wednesday, August 3, 2016 6:10 PM
To: Rychnovsky, Dusan
Cc: user@spark.apache.org
Subject: Re: Managed memory leak detected + OutOfMemoryError:
his PR is for the branch-1.6
version of the commits PR #10024. In #9241 It implemented a mechanism to call
spill() on those SQL operators that sup...
Read more...<https://github.com/apache/spark/pull/13027/>
____
From: Rychnovsky, Dusan
Sent: Wednesday, August 3,
I have 1.6.0 and
therefore should have it fixed, right? Or what do I do to fix it?
Thanks,
Dusan
From: Ted Yu <yuzhih...@gmail.com>
Sent: Wednesday, August 3, 2016 3:52 PM
To: Rychnovsky, Dusan
Cc: user@spark.apache.org
Subject: Re: Managed memory leak detected
Hi,
I have a Spark workflow that when run on a relatively small portion of data
works fine, but when run on big data fails with strange errors. In the log
files of failed executors I found the following errors:
Firstly
> Managed memory leak detected; size = 263403077 bytes, TID = 6524
And
Hi,
I am trying to launch my Spark application from within my Java application via
the SparkSubmit class, like this:
List args = new ArrayList<>();
args.add("--verbose");
args.add("--deploy-mode=cluster");
args.add("--master=yarn");
...
SparkSubmit.main(args.toArray(new
Hi,
can somebody please explain the way FullOuterJoin works on Spark? Does each
intersection get fully loaded to memory?
My problem is as follows:
I have two large data-sets:
* a list of web pages,
* a list of domain-names with specific rules for processing pages from that
domain.
I am
