using Spark Kmeans. Any
solutions from the experts. Would be really helpful.
val kMeans = new KMeans().setK(reductionCount).setMaxIter(30)
val kMeansModel = kMeans.fit(df)
Error is occured when calling kmeans.fit
Exception in thread "main" java.lang.OutOfMemoryError: Java
;main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.spark.mllib.linalg.SparseVector.toArray(Vectors.scala:760)
at
org.apache.spark.mllib.clustering.VectorWithNorm.toDense(KMeans.scala:614)
at
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParall
nt
val tmp = graph.triangleCount().vertices.filter{ case (vid, count) => count >
0 }
var numberOfTriangles = tmp.map(a => a._2).sum()
Even though it manages to compute the number of triplets, I can’t compute the
number of triangles. Every time I get an exception OOM - Java Heap Space on
s
n in finally: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
~[na:1.8.0_162]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_162]
at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadca
of row object increased from 20kb to about 4mb. Now the
same code runs into java heap space issues and application is shut down
with an out of memory exception.
Seems dataframe sort operations cannot handle large objects. I took an
heap dump and saw an large array-of-array. I would expect such object
that is being created is huge. Could it be due to
that?
Thanks!
Vinay
18/02/17 00:59:02 ERROR Utils: throw uncaught fatal error in thread
SparkListenerBus
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.(String.java:207
ang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3230)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArr
May be you can repartition?
2017-09-04 9:25 GMT+08:00 KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>:
> Hi,
>
> I am getting java.lang.OutOfMemoryError: Java heap space error whenever I
> ran the spark sql job.
>
> I came to conclusion issue is because of reading numb
Hi,
I am getting java.lang.OutOfMemoryError: Java heap space error whenever I
ran the spark sql job.
I came to conclusion issue is because of reading number of files from spark.
I am reading 37 partitions and each partition has around 2000 files with
filesize more than 128 MB 37*2000 files
10:09:26 INFO BlockManagerInfo: Removed taskresult_362 on
ip-...-45.dev:40963 in memory (size: 5.2 MB, free: 8.9 GB)
17/04/24 10:09:26 INFO TaskSetManager: Finished task 125.0 in stage 1.0
(TID 359) in 4383 ms on ip-...-45.dev (125/234)
#
# java.lang.OutOfMemoryError: Java heap space
Hi,
I have 1 master and 4 slave node. Input data size is 14GB.
Slave Node config : 32GB Ram,16 core
I am trying to train word embedding model using spark. It is going out of
memory. To train 14GB of data how much memory do i require?.
I have givem 20gb per executor but below shows it is using
Hi,
Need a help to figure out and solve heap space problem.
I have query which contains 15+ table and when i trying to print out the
result(Just 23 rows) it throws heap space error.
Following command i tried in standalone mode:
(My mac machine having 8 core and 15GB ram)
;>
>>>>>>> HTH
>>>>>>>
>>>>>>> Dr Mich Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> LinkedIn *
>>>>>>> https://www.linkedin.c
;>> LinkedIn *
>>>>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>&
t;
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>&
truction.
>>>>
>>>>
>>>>
>>>> On 24 October 2016 at 12:19, sankarmittapally <
>>>> sankar.mittapa...@creditvidya.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a three node cluster
node cluster with 30G of Memory. I am trying to
>>>> analyzing
>>>> the data of 200MB and running out of memory every time. This is the
>>>> command
>>>> I am using
>>>>
>>>> Driver Memory = 10G
>>>> Executor mem
> "spark://ip-172-31-6-116:7077",sparkConfig=list(spark.execut
>>> or.memory="10g",spark.app.name="Testing",spark.driver.memory
>>> ="14g",spark.executor.extraJavaOption="-Xms2g
>>
t;10g",spark.app.name="Testing",spark.driver.
>> memory="14g",spark.executor.extraJavaOption="-Xms2g
>> -Xmx5g -XX:MaxPermSize=1024M",spark.driver.extraJavaOption="-Xms2g -Xmx5g
>> -XX:MaxPermSize=1024M",spark.cores.max="2"))
&g
> -Xmx5g -XX:MaxPermSize=1024M",spark.driver.extraJavaOption="-Xms2g -Xmx5g
> -XX:MaxPermSize=1024M",spark.cores.max="2"))
>
>
> [D 16:43:51.437 NotebookApp] 200 GET
> /api/contents?type=directory&_=1477289197671 (123.176.38.226) 7.96ms
> Exception in thre
ot;))
[D 16:43:51.437 NotebookApp] 200 GET
/api/contents?type=directory&_=1477289197671 (123.176.38.226) 7.96ms
Exception in thread "broadcast-exchange-0" java.lang.OutOfMemoryError: Java
heap space
ead "dispatcher-event-loop-1"
java.lang.OutOfMemoryError: Java heap space
> How much heap memory do you give the driver ?
>
> On Fri, Jul 22, 2016 at 2:17 PM, Andy Davidson <a...@santacruzintegration.com>
> wrote:
>> Given I get a stack trace in my python notebook I am
TaskSetManager: Stage 146 contains a task of very
> large size (145 KB). The maximum recommended task size is 100 KB.
>
> 16/07/22 18:39:47 WARN HeartbeatReceiver: Removing executor 2 with no
> recent heartbeats: 153037 ms exceeds timeout 12 ms
>
> Excepti
java.lang.OutOfMemoryError:
Java heap space
at java.util.jar.Manifest$FastInputStream.(Manifest.java:332)
at java.util.jar.Manifest$FastInputStream.(Manifest.java:327)
at java.util.jar.Manifest.read(Manifest.java:195)
at java.util.jar.Manifest.(Manifest.java:69)
at java.util.jar.JarFile.getManif
GB Ubuntu server...
>>>>>
>>>>> I have changed things in the conf file, but it looks like Spark does not
>>>>> care, so I wonder if my issues are with the driver or executor.
>>>>>
>>>>> I set:
>>>
ues are with the driver or executor.
>>>>
>>>> I set:
>>>>
>>>> spark.driver.memory 20g
>>>> spark.executor.memory 20g
>>>> And, whatever I do, the crash is always at the same spot in the app, which
>>>
:
>>>
>>> spark.driver.memory 20g
>>> spark.executor.memory 20g
>>> And, whatever I do, the crash is always at the same spot in the app, which
>>> makes me think that it is a driver pro
ame spot in the app, which
>> makes me think that it is a driver problem.
>>
>> The exception I get is:
>>
>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208,
>> micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>> at java.nio.H
gt;
> The exception I get is:
>
> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208,
> micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapCharBuffer.(HeapCharBuffer.java:57)
> at java.n
): java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapCharBuffer.(HeapCharBuffer.java:57)
at java.nio.CharBuffer.allocate(CharBuffer.java:335)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
at org.apache.hadoop.io.Text.decode(Text.java:412
Thanks Ted for the input. I was able to get it working with pyspark shell
but the same job submitted via 'spark-submit' using client or cluster
deploy mode ends up with these errors:
~
java.lang.OutOfMemoryError: Java heap space
at java.lang.Object.clone(Native Method
skowski
>>
>>
>> On Sat, Apr 9, 2016 at 7:51 PM, Buntu Dev <buntu...@gmail.com> wrote:
>> > I'm running this motif pattern against 1.5M vertices (5.5mb) and 10M
>> (60mb)
>> > edges:
>> >
>> > tgraph.find("(a)-[]->(b);
gt; I'm running this motif pattern against 1.5M vertices (5.5mb) and 10M
> (60mb)
> > edges:
> >
> > tgraph.find("(a)-[]->(b); (c)-[]->(b); (c)-[]->(d)")
> >
> > I keep running into Java heap space errors:
> >
> > ~
> >
es (5.5mb) and 10M (60mb)
> edges:
>
> tgraph.find("(a)-[]->(b); (c)-[]->(b); (c)-[]->(d)")
>
> I keep running into Java heap space errors:
>
> ~
>
> ERROR actor.ActorSystemImpl: Uncaught fatal error from thread
> [sparkDriver-akka.actor.defau
I'm running this motif pattern against 1.5M vertices (5.5mb) and 10M (60mb)
edges:
tgraph.find("(a)-[]->(b); (c)-[]->(b); (c)-[]->(d)")
I keep running into Java heap space errors:
~
ERROR actor.ActorSystemImpl: Uncaught fatal error from thread
[sparkDriver-akka.actor.de
ActorSystem [sparkDriver]
java.lang.OutOfMemoryError: Java heap space
at
com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
at
akka.remote.transport.AkkaPduProtobufCodec$.constructMessage(AkkaPduCodec.scala:138)
at akka.remote.EndpointWriter.writeSend(Endpoint.scala:740
t;
>>
>> *From:* Shuai Zheng [mailto:szheng.c...@gmail.com]
>> *Sent:* Wednesday, November 04, 2015 3:22 PM
>> *To:* user@spark.apache.org
>> *Subject:* [Spark 1.5]: Exception in thread "broadcast-hash-join-2"
>> java.lang.OutOfMemoryError: Java heap
s proven
>>> that there is no issue on the logic and data, it is caused by the new
>>> version of Spark.
>>>
>>>
>>>
>>> So I want to know any new setup I should set in Spark 1.5 to make it
>>> work?
>>>
>>>
>>>
>>> R
oin-2"
java.lang.OutOfMemoryError: Java heap space
Hi All,
I have a program which actually run a bit complex business (join) in spark.
And I have below exception:
I running on Spark 1.5, and with parameter:
spark-submit --deploy-mode client --executor-cores=24 --driver-memory=2G
uot;).set("spark.sql.autoBroadcastJoinThreshold",
"104857600");
This is running on AWS c3*8xlarge instance. I am not sure what kind of
parameter I should set if I have below OutOfMemoryError exception.
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9
te all the
> productionName and product catagory together, without even consider removing
> duplication.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap
the stuff without String operations?
>
> On 24 September 2015 at 10:11, java8964 <java8...@hotmail.com
> <mailto:java8...@hotmail.com>> wrote:
> Try to increase partitions count, that will make each partition has less data.
>
> Yong
>
> Subject: Re: Java Heap Space
rtitioner of "userid".
Can you show us the query after you add "regex" and "concatenation"?
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 15:34:48 +0300
CC: user@spark.apache.org
To: jingyu.zh...@news.com.au; java8...@hotma
ns simply depends on the Hash partitioner of "userid".
>
> Can you show us the query after you add "regex" and "concatenation"?
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 15:34:48 +0300
&
ion.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 18:56:51 +0300
> CC: jingyu.
.
But both query still should push similar records count per partition, but with
much of different volume size of data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 18:56:51 +0300
CC: jingyu.zh...@news.com.au; user@spark.apache.org
To: java8...@hotmail.com
Yes
ans that one
> partition having way more data than the rest of partitions.
>
> Yong
>
> From: yu...@useinsider.com
> Subject: Java Heap Space Error
> Date: Wed, 23 Sep 2015 23:07:17 +0300
> To: user@spark.apache.org
>
> What can cause this issue in the attached pic
Try to increase partitions count, that will make each partition has less data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 00:32:47 +0300
CC: user@spark.apache.org
To: java8...@hotmail.com
Yes, it’s possible. I use S3 as data source. My external
ta.
>
> Yong
>
> ------
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 00:32:47 +0300
> CC: user@spark.apache.org
> To: java8...@hotmail.com
>
>
> Yes, it’s possible. I use S3 as data source. My extern
, which led to the dreaded Java heap space error. See the
stack trace at the end of this message.
When this happens, I see 10's of executors in EXITED state, a couple in
LOADING and one in RUNNING. All of them are retrying the same task all
over again, and keep failing with the same Java heap space
@spark.apache.org
Sent: Thursday, June 11, 2015 8:43 AM
Subject: spark-sql from CLI ---EXCEPTION: java.lang.OutOfMemoryError: Java
heap space
hey guys
Using Hive and Impala daily intensively.Want to transition to spark-sql in CLI
mode
Currently in my sandbox I am using the Spark (standalone mode
event ([id: 0x01b99855,
/10.0.0.19:58117 = /10.0.0.19:52016] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42
---EXCEPTION: java.lang.OutOfMemoryError:
Java heap space
It sounds like this might be caused by a memory configuration problem. In
addition to looking at the executor memory, I'd also bump up the driver memory,
since it appears that your shell is running out of memory when collecting a
large query
:58117 = /10.0.0.19:52016] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.init
while handling an exception event ([id: 0x01b99855,
/10.0.0.19:58117 = /10.0.0.19:52016] EXCEPTION: java.lang.OutOfMemoryError:
Java heap space)
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42
: 0x01b99855,
/10.0.0.19:58117 = /10.0.0.19:52016] EXCEPTION: java.lang.OutOfMemoryError:
Java heap space)
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.init
handler while handling an exception event ([id: 0x01b99855,
/10.0.0.19:58117 = /10.0.0.19:52016] EXCEPTION: java.lang.OutOfMemoryError:
Java heap space)java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42
Hi,
I'm trying to train an SVM on KDD2010 dataset (available from libsvm). But
I'm getting java.lang.OutOfMemoryError: Java heap space error. The dataset
is really sparse and have around 8 million data points and 20 million
features. I'm using a cluster of 8 nodes (each with 8 cores and 64G RAM
Try increasing your driver memory.
Thanks
Best Regards
On Thu, Apr 16, 2015 at 6:09 PM, sarath sarathkrishn...@gmail.com wrote:
Hi,
I'm trying to train an SVM on KDD2010 dataset (available from libsvm). But
I'm getting java.lang.OutOfMemoryError: Java heap space error. The
dataset
)
15/04/07 18:11:03 ERROR Utils: Uncaught exception in thread
task-result-getter-0
java.lang.OutOfMemoryError: Java heap space
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:61)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985
:49451 (size: 163.7 MB)
15/04/07 18:11:03 INFO TaskSetManager: Starting task 30.0 in stage 1.0
(TID 32, cloud2, NODE_LOCAL, 1296 bytes)
15/04/07 18:11:03 ERROR Utils: Uncaught exception in thread
task-result-getter-0
java.lang.OutOfMemoryError: Java heap space
-Dspark.deploy.defaultCores=. When you have not
enough memory, reduce the concurrency of your executor, it will lower the
memory requirement, with running in a slower speed.
Yong
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from HDFS
From: lidali
in a slower speed.
Yong
--
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from
HDFS
From: lidali...@gmail.com
To: user@spark.apache.org
Any help?please.
Help me do a right configure.
李铖 lidali
Hi all,
sometimes you can see OutOfMemoryException: Java heap space of executor in
Spark. There many ideas about how to work arounds.
My question is: how does executor execute tasks from the point of view of
memory usage and parallelism?
Picture in my mind is:
Executor is JVM instance. Number
Mem=270794224, maxMem=311387750
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.init(Unknown Source)
at
org.apache.spark.api.python.PythonRDD$$anon$2.run(PythonRDD.scala:62)
14/07/09 01:46:15 INFO MemoryStore: Block rdd_2212_0 stored as values
.1001560.n3.nabble.com/KMeans-with-large-clusters-Java-Heap-Space-tp21432p22153.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional
the memory is sufficient, spark.executor.memory 30GB )
15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 (TID
7653)
java.lang.OutOfMemoryError: Java heap space
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410
)
15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0
(TID 7653)
java.lang.OutOfMemoryError: Java heap space
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410)
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.put
much!
(I think the memory is sufficient, spark.executor.memory 30GB )
15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 (TID
7653)
java.lang.OutOfMemoryError: Java heap space
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize
: Exception in task 162.0 in stage 719.0 (TID
7653)
java.lang.OutOfMemoryError: Java heap space
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410)
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113
the following errors during computation(graphx), anyone has
ideas on this? thanks so much!
(I think the memory is sufficient, spark.executor.memory 30GB )
15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0
(TID 7653)
java.lang.OutOfMemoryError: Java heap space
), anyone has
ideas on this? thanks so much!
(I think the memory is sufficient, spark.executor.memory 30GB )
15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0
(TID 7653)
java.lang.OutOfMemoryError: Java heap space
heap space
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410)
at
com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113)
at
com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject
], k: Int): KMeansModel = {
KMeans.train(raw, k, embeddingNames = List(LOW_DIMENSIONAL_RI)
}
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/KMeans-with-large-clusters-Java-Heap-Space-tp21432p21437.html
Sent from the Apache Spark User List mailing list
getting Java Heap
Space error. Working with 3 nodes cluster with each 8 GB memory and 2 cores.
Played with different configuration, but no luck...
what am I missing any suggestions?
here is my code
val sparkConf = new SparkConf().setMaster(spark://master:7077)
.setAppName(SparkKMeans)
.set
I'm trying to process a large dataset, mapping/filtering works ok, but
as long as I try to reduceByKey, I get out of memory errors:
http://pastebin.com/70M5d0Bn
Any ideas how I can fix that?
Thanks.
-
To unsubscribe, e-mail:
Hi Kane-
http://spark.apache.org/docs/latest/tuning.html has excellent information that
may be helpful. In particular increasing the number of tasks may help, as well
as confirming that you don’t have more data than you're expecting landing on a
key.
Also, if you are using spark 1.2.0,
(spark.storage.memoryFraction,0.3)//default is 0.6
I have to set rank value under 40, otherwise occure this problem.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20755.html
Sent from the Apache
) //default is 0.2
conf.set(spark.storage.memoryFraction,0.3)//default is 0.6
I have to set rank value under 40, otherwise occure this problem.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space
.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20714.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
How many working nodes do these 100 executors locate at?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20610.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
:34602], 1 messages pending
14/10/20 22:38:41 INFO ConnectionManager: Accepted connection from
[cse-hadoop-113/192.168.0.113]
Exception in thread pool-5-thread-3 java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.init(HeapByteBuffer.java:57
\*
* --conf spark.driver.memory=2g \*
* target/scala-2.10/my-job_2.10-1.0.jar*
I get the following error :
*Exception in thread stdin writer for List(patch_matching_similarity)
java.lang.OutOfMemoryError: Java heap space*
* at java.util.Arrays.copyOf(Arrays.java:2271
] \
--conf spark.executor.memory=4g \
--conf spark.driver.memory=2g \
target/scala-2.10/my-job_2.10-1.0.jar
I get the following error :
Exception in thread stdin writer for List(patch_matching_similarity)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf
) associated
with image id. My goal is to draw these primitives on the corresponding
image. So my first attempt is to join images and primitives by image ids
and then do the drawing.
But, when I do
*primitives.join(images) *
I got the following error :
*java.lang.OutOfMemoryError: Java heap space
the underlying
cause. I don't have a copy of the stack trace handy but can recreate it if
necessary. It was somewhere in the init for HeapByteBuffer. Any advice
would be helpful.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Driver-OOME-Java-heap-space
the underlying
cause. I don't have a copy of the stack trace handy but can recreate it if
necessary. It was somewhere in the init for HeapByteBuffer. Any advice
would be helpful.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Driver-OOME-Java-heap
Thanks Marcelo, that looks like the same thing. I'll follow the Jira ticket
for updates.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Driver-OOME-Java-heap-space-when-executors-request-map-output-locations-tp13827p13829.html
Sent from the Apache
Marcelo, that looks like the same thing. I'll follow the Jira ticket
for updates.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Driver-OOME-Java-heap-space-when-executors-request-map-output-locations-tp13827p13829.html
Sent from the Apache Spark
]
java.lang.OutOfMemoryError: Java heap space
at
com.google.protobuf_spark.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
at
akka.remote.transport.AkkaPduProtobufCodec$.constructPayload(AkkaPduCodec.scala:145)
at
akka.remote.transport.AkkaProtocolHandle.write
ERROR actor.ActorSystemImpl: Uncaught fatal error from
thread [spark-akka.actor.default-dispatcher-20] shutting down ActorSystem
[spark]
java.lang.OutOfMemoryError: Java heap space
at
com.google.protobuf_spark.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62
:
Started 0 remote fetches in 1 ms14/07/31 09:48:17 ERROR Executor: Exception in
task ID 5java.lang.OutOfMemoryError: Java heap space at
java.util.Arrays.copyOf(Arrays.java:2271)at
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178
: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
at org.apache.spark.executor.Executor$TaskRunner.run
.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.init(HeapChannelBuffer.java:42)
---
The specifics of the job is as follows:
- It reads 168016 files on the HDFS, by calling
sc.textFile(hdfs://cluster01/user/data/*/*/*.csv)
- The total size of the files is 164,111,123,686
.
-Original Message-
From: innowireless TaeYun Kim [mailto:taeyun@innowireless.co.kr]
Sent: Wednesday, July 02, 2014 5:58 PM
To: user@spark.apache.org
Subject: Help: WARN AbstractNioSelector: Unexpected exception in the
selector loop. java.lang.OutOfMemoryError: Java heap space
Hi
. java.lang.OutOfMemoryError: Java heap space
Also, the machine on which the driver program runs constantly uses about
7~8% of 100Mbps network connection.
Is the driver program involved in the reduceByKey() somehow?
BTW, currently an accumulator is used, but the network usage does not drop
even when accumulator
-java-lang-outOfMemoryError-Java-Heap-Space-td7735.html#a7736)
changing the number of partitions in my RDD by using coalesce(8) and the
error persisted
b) I tried changing SPARK_WORKER_MEM=2g, SPARK_EXECUTOR_MEMORY=10g,
and both did not work
c) I strongly suspect there is a class path error
:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183)
at
org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2244)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280
folder,
and I get the following error:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183)
at
org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2244
the following error:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183)
at
org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2244)
at
org.apache.hadoop.io.ObjectWritable.readObject
1 - 100 of 110 matches
Mail list logo
