Re: Connection Reset by Peer : failed to remove cached rdd

[Harsh Sharma]

[Stage 284:>(199 + 1) / 200][Stage 292:>  (1 + 3) / 200]
[Stage 284:>(199 + 1) / 200][Stage 292:>  (2 + 3) / 200]
[Stage 292:>  (2 + 4) / 
200][14/06/21 10:46:17,006 WARN  shuffle-server-4](TransportChannelHandler) 
Exception in connection from 
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378)
at 
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at 
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
[14/06/21 10:46:17,010 ERROR shuffle-server-4](TransportResponseHandler) Still 
have 1 requests outstanding when connection from  is closed 
[14/06/21 10:46:17,012 ERROR Spark Context Cleaner](ContextCleaner) Error 
cleaning broadcast 159 
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378)
at 
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at 
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
[14/06/21 10:46:17,012 WARN  
block-manager-ask-thread-pool-69](BlockManagerMaster) Failed to remove 
broadcast 159 with removeFromMaster = true - Connection reset by peer 
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378)
at 
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at 
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)

On 2021/07/29 12:46:01, 
Big data developer need help relat to spark gateway roles in 2.0
  wrote: 
> Hi Team ,
> 
> We are facing issue in production where we are getting frequent
> 
> Still have 1 request outstanding when connection with the hostname was closed
> 
> connection reset by peer : errors as well as warnings : failed to remove cache
> rdd or failed to remove broadcast variable.
> 
> Please help us how 

Re: Connection Reset by Peer : failed to remove cached rdd

[Artemis User]

Can you please post the error log/exception messages?  There is not 
enough info to help diagnose what the real problem is


On 7/29/21 8:55 AM, Big data developer need help relat to spark gateway 
roles in 2.0 wrote:


Hi Team ,

We are facing issue in production where we are getting frequent

Still have 1 request outstanding when connection with the hostname was 
closed


connection reset by peer : errors as well as warnings  : failed to 
remove cache rdd or failed  to remove broadcast variable.


Please help us how to mitigate this  :

Executor memory : 12g

Network timeout :   60

Heartbeat interval : 25


 
	Virus-free. www.avast.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
- 
To unsubscribe e-mail: user-unsubscr...@spark.apache.org 

Connection Reset by Peer : failed to remove cached rdd

[Big data developer need help relat to spark gateway roles in 2 . 0]

Hi Team , We are facing issue in production where we are getting frequent Still have 1 request outstanding when connection with the hostname was closed connection reset by peer : errors as well as warnings  : failed to remove cache rdd or failed  to remove broadcast variable. Please help us how to mitigate this  : Executor memory : 12g Network timeout :   60Heartbeat interval : 25

	

		Virus-free. www.avast.com
		
	
 

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Connection Reset by Peer : failed to remove cached rdd

[Big data developer need help relat to spark gateway roles in 2 . 0]

Hi Team , We are facing issue in production where we are getting frequent Still have 1 request outstanding when connection with the hostname was closed connection reset by peer : errors as well as warnings  : failed to remove cache rdd or failed  to remove broadcast variable. Please help us how to mitigate this  : Executor memory : 12g Network timeout :   60Heartbeat interval : 25  

	

		Virus-free. www.avast.com
		
	
 

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Reading Back a Cached RDD

[aka.fe2s]

Nick, what is your use-case?


On Thu, Mar 24, 2016 at 11:55 PM, Marco Colombo  wrote:

> You can persist off-heap, for example with tachyon, now called Alluxio.
> Take a look at off heap peristance
>
> Regards
>
>
> Il giovedì 24 marzo 2016, Holden Karau  ha scritto:
>
>> Even checkpoint() is maybe not exactly what you want, since if reference
>> tracking is turned on it will get cleaned up once the original RDD is out
>> of scope and GC is triggered.
>> If you want to share persisted RDDs right now one way to do this is
>> sharing the same spark context (using something like the spark job server
>> or IBM Spark Kernel).
>>
>> On Thu, Mar 24, 2016 at 11:28 AM, Nicholas Chammas <
>> nicholas.cham...@gmail.com> wrote:
>>
>>> Isn’t persist() only for reusing an RDD within an active application?
>>> Maybe checkpoint() is what you’re looking for instead?
>>> ​
>>>
>>> On Thu, Mar 24, 2016 at 2:02 PM Afshartous, Nick <
>>> nafshart...@turbine.com> wrote:
>>>

 Hi,


 After calling RDD.persist(), is then possible to come back later and
 access the persisted RDD.

 Let's say for instance coming back and starting a new Spark shell
 session.  How would one access the persisted RDD in the new shell session ?


 Thanks,

 --

Nick

>>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
> --
> Ing. Marco Colombo
>



-- 
--
Oleksiy Dyagilev

Re: Reading Back a Cached RDD

[Marco Colombo]

You can persist off-heap, for example with tachyon, now called Alluxio.
Take a look at off heap peristance

Regards

Il giovedì 24 marzo 2016, Holden Karau  ha scritto:

> Even checkpoint() is maybe not exactly what you want, since if reference
> tracking is turned on it will get cleaned up once the original RDD is out
> of scope and GC is triggered.
> If you want to share persisted RDDs right now one way to do this is
> sharing the same spark context (using something like the spark job server
> or IBM Spark Kernel).
>
> On Thu, Mar 24, 2016 at 11:28 AM, Nicholas Chammas <
> nicholas.cham...@gmail.com
> > wrote:
>
>> Isn’t persist() only for reusing an RDD within an active application?
>> Maybe checkpoint() is what you’re looking for instead?
>> ​
>>
>> On Thu, Mar 24, 2016 at 2:02 PM Afshartous, Nick > > wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>> After calling RDD.persist(), is then possible to come back later and
>>> access the persisted RDD.
>>>
>>> Let's say for instance coming back and starting a new Spark shell
>>> session.  How would one access the persisted RDD in the new shell session ?
>>>
>>>
>>> Thanks,
>>>
>>> --
>>>
>>>Nick
>>>
>>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>


-- 
Ing. Marco Colombo

Re: Reading Back a Cached RDD

[Holden Karau]

Even checkpoint() is maybe not exactly what you want, since if reference
tracking is turned on it will get cleaned up once the original RDD is out
of scope and GC is triggered.
If you want to share persisted RDDs right now one way to do this is sharing
the same spark context (using something like the spark job server or IBM
Spark Kernel).

On Thu, Mar 24, 2016 at 11:28 AM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:

> Isn’t persist() only for reusing an RDD within an active application?
> Maybe checkpoint() is what you’re looking for instead?
> ​
>
> On Thu, Mar 24, 2016 at 2:02 PM Afshartous, Nick 
> wrote:
>
>>
>> Hi,
>>
>>
>> After calling RDD.persist(), is then possible to come back later and
>> access the persisted RDD.
>>
>> Let's say for instance coming back and starting a new Spark shell
>> session.  How would one access the persisted RDD in the new shell session ?
>>
>>
>> Thanks,
>>
>> --
>>
>>Nick
>>
>


-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

Re: Reading Back a Cached RDD

[Nicholas Chammas]

Isn’t persist() only for reusing an RDD within an active application? Maybe
checkpoint() is what you’re looking for instead?
​

On Thu, Mar 24, 2016 at 2:02 PM Afshartous, Nick 
wrote:

>
> Hi,
>
>
> After calling RDD.persist(), is then possible to come back later and
> access the persisted RDD.
>
> Let's say for instance coming back and starting a new Spark shell
> session.  How would one access the persisted RDD in the new shell session ?
>
>
> Thanks,
>
> --
>
>Nick
>

Reading Back a Cached RDD

[Afshartous, Nick]

Hi,


After calling RDD.persist(), is then possible to come back later and access the 
persisted RDD.

Let's say for instance coming back and starting a new Spark shell session.  How 
would one access the persisted RDD in the new shell session ?


Thanks,

--

   Nick

Re: when cached RDD will unpersist its data

[eric wong]

In a case that memory cannot hold all the cached RDD, then BlockManager
will evict some older block for storage of new RDD block.


Hope that will helpful.

2015-06-24 13:22 GMT+08:00 bit1...@163.com bit1...@163.com:

 I am kind of consused about when cached RDD will unpersist its data. I
 know we can explicitly unpersist it with RDD.unpersist ,but can it be
 unpersist automatically by the spark framework?
 Thanks.

 --
 bit1...@163.com




-- 
王海华

when cached RDD will unpersist its data

[bit1...@163.com]

I am kind of consused about when cached RDD will unpersist its data. I know we 
can explicitly unpersist it with RDD.unpersist ,but can it be unpersist 
automatically by the spark framework?
Thanks.



bit1...@163.com

Re: set spark.storage.memoryFraction to 0 when no cached RDD and memory area for broadcast value?

[Akhil Das]

You could try leaving all the configuration values to default and running
your application and see if you are still hitting the heap issue, If so try
adding a Swap space to the machines which will definitely help. Another way
would be to set the heap space manually (export _JAVA_OPTIONS=-Xmx5g)

Thanks
Best Regards

On Wed, Apr 8, 2015 at 12:45 AM, Shuai Zheng szheng.c...@gmail.com wrote:

 Hi All,



 I am a bit confused on spark.storage.memoryFraction, this is used to set
 the area for RDD usage, will this RDD means only for cached and persisted
 RDD? So if my program has no cached RDD at all (means that I have no
 .cache() or .persist() call on any RDD), then I can set this
 spark.storage.memoryFraction to a very small number or even zero?



 I am writing a program which consume a lot of memory (broadcast value,
 runtime, etc). But I have no cached RDD, so should I just turn off this
 spark.storage.memoryFraction to 0 (which will help me to improve the
 performance)?



 And I have another issue on the broadcast, when I try to get a broadcast
 value, it throws me out of memory error, which part of memory should I
 allocate more (if I can’t increase my overall memory size).



 java.lang.OutOfMemoryError: Java heap spac

 e

 at
 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

 rraySerializer.read(DefaultArraySerializers.java:218)

 at
 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

 rraySerializer.read(DefaultArraySerializers.java:200)

 at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

 d(FieldSerializer.java:611)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

 lizer.java:221)

 at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

 d(FieldSerializer.java:605)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

 lizer.java:221)

 at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)

 at
 org.apache.spark.serializer.KryoDeserializationStream.readObject(Kryo

 Serializer.scala:138)

 at
 org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Ser

 ializer.scala:133)

 at
 org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)

 at
 org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:2

 48)

 at
 org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:13

 6)

 at
 org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:5

 49)

 at
 org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:431

 )

 at
 org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlo

 ck$1.apply(TorrentBroadcast.scala:167)

 at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152)

 at
 org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(Torren

 tBroadcast.scala:164)

 at
 org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(Torrent

 Broadcast.scala:64)

 at
 org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.s

 cala:64)

 at
 org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast

 .scala:87)





 Regards,



 Shuai

Re: set spark.storage.memoryFraction to 0 when no cached RDD and memory area for broadcast value?

[twinkle sachdeva]

Hi,

In one of the application we have made which had no clone stuff, we have
set the value of spark.storage.memoryFraction to very low, and yes that
gave us performance benefits.

Regarding that issue, you should also look at the data you are trying to
broadcast, as sometimes creating that data structure at executor's itself
as singleton helps.

Thanks,


On Tue, Apr 14, 2015 at 12:23 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:

 You could try leaving all the configuration values to default and running
 your application and see if you are still hitting the heap issue, If so try
 adding a Swap space to the machines which will definitely help. Another way
 would be to set the heap space manually (export _JAVA_OPTIONS=-Xmx5g)

 Thanks
 Best Regards

 On Wed, Apr 8, 2015 at 12:45 AM, Shuai Zheng szheng.c...@gmail.com
 wrote:

 Hi All,



 I am a bit confused on spark.storage.memoryFraction, this is used to set
 the area for RDD usage, will this RDD means only for cached and persisted
 RDD? So if my program has no cached RDD at all (means that I have no
 .cache() or .persist() call on any RDD), then I can set this
 spark.storage.memoryFraction to a very small number or even zero?



 I am writing a program which consume a lot of memory (broadcast value,
 runtime, etc). But I have no cached RDD, so should I just turn off this
 spark.storage.memoryFraction to 0 (which will help me to improve the
 performance)?



 And I have another issue on the broadcast, when I try to get a broadcast
 value, it throws me out of memory error, which part of memory should I
 allocate more (if I can’t increase my overall memory size).



 java.lang.OutOfMemoryError: Java heap spac

 e

 at
 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

 rraySerializer.read(DefaultArraySerializers.java:218)

 at
 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

 rraySerializer.read(DefaultArraySerializers.java:200)

 at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

 d(FieldSerializer.java:611)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

 lizer.java:221)

 at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

 d(FieldSerializer.java:605)

 at
 com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

 lizer.java:221)

 at
 com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)

 at
 org.apache.spark.serializer.KryoDeserializationStream.readObject(Kryo

 Serializer.scala:138)

 at
 org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Ser

 ializer.scala:133)

 at
 org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)

 at
 org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:2

 48)

 at
 org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:13

 6)

 at
 org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:5

 49)

 at
 org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:431

 )

 at
 org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlo

 ck$1.apply(TorrentBroadcast.scala:167)

 at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152)

 at
 org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(Torren

 tBroadcast.scala:164)

 at
 org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(Torrent

 Broadcast.scala:64)

 at
 org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.s

 cala:64)

 at
 org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast

 .scala:87)





 Regards,



 Shuai

set spark.storage.memoryFraction to 0 when no cached RDD and memory area for broadcast value?

[Shuai Zheng]

Hi All,

 

I am a bit confused on spark.storage.memoryFraction, this is used to set the
area for RDD usage, will this RDD means only for cached and persisted RDD?
So if my program has no cached RDD at all (means that I have no .cache() or
.persist() call on any RDD), then I can set this
spark.storage.memoryFraction to a very small number or even zero?

 

I am writing a program which consume a lot of memory (broadcast value,
runtime, etc). But I have no cached RDD, so should I just turn off this
spark.storage.memoryFraction to 0 (which will help me to improve the
performance)?

 

And I have another issue on the broadcast, when I try to get a broadcast
value, it throws me out of memory error, which part of memory should I
allocate more (if I can't increase my overall memory size).

 

java.lang.OutOfMemoryError: Java heap spac

e

at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

rraySerializer.read(DefaultArraySerializers.java:218)

at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA

rraySerializer.read(DefaultArraySerializers.java:200)

at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699)

at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

d(FieldSerializer.java:611)

at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

lizer.java:221)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)

at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea

d(FieldSerializer.java:605)

at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria

lizer.java:221)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)

at
org.apache.spark.serializer.KryoDeserializationStream.readObject(Kryo

Serializer.scala:138)

at
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Ser

ializer.scala:133)

at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)

at
org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:2

48)

at
org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:13

6)

at
org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:5

49)

at
org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:431

)

at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlo

ck$1.apply(TorrentBroadcast.scala:167)

at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152)

at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(Torren

tBroadcast.scala:164)

at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(Torrent

Broadcast.scala:64)

at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.s

cala:64)

at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast

.scala:87)

 

 

Regards,

 

Shuai

Re: How to get the cached RDD

[praveenbalaji]

sc.getPersistentRDDs(0).asInstanceOf[RDD[Array[Double]]]



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-get-the-cached-RDD-tp22114p22122.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Access time to an elemnt in cached RDD

[Sean Owen]

It may involve access an element of an RDD from a remote machine and
copying it back to the driver. That and the small overhead of job
scheduling could be a millisecond.

You're comparing to just reading an entry from memory, which is of
course faster.

I don't think you should think of an RDD as something you query at
scale in real-time. It's not a NoSQL store.

On Mon, Feb 23, 2015 at 6:02 PM, shahab shahab.mok...@gmail.com wrote:
 Hi,

 I just wonder what would be the access time to take one element from a
 cached RDD? if I have understood correctly, access to RDD elements is not as
 fast as accessing e.g. HashMap and it could take up to  mili seconds compare
 to nano seconds in HashMap, which is quite significant difference if you
 plan for near real-time response from Spark ?!

 best,

 /Shahab



-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Access time to an elemnt in cached RDD

[shahab]

Hi,

I just wonder what would be the access time to take one element from a
cached RDD? if I have understood correctly, access to RDD elements is not
as fast as accessing e.g. HashMap and it could take up to  mili seconds
compare to nano seconds in HashMap, which is quite significant difference
if you plan for near real-time response from Spark ?!

best,

/Shahab

Re: Why cached RDD is recomputed again?

[shahab]

Thanks Sean, but I don't think that fitting into memory  is the case,
because:
1- I can see in the UI that 100% of RDD is cached, (moreover the RDD is
quite small, 100 MB, while worker has 1.5 GB)
2- I also tried  MEMORY_AND_DISK, but absolutely no difference !

Probably I have messed up somewhere else!
Do you have any other idea where I should look for the cause?

best,
/Shahab

On Wed, Feb 18, 2015 at 4:22 PM, Sean Owen so...@cloudera.com wrote:

 The mostly likely explanation is that you wanted to put all the
 partitions in memory and they don't all fit. Unless you asked to
 persist to memory or disk, some partitions will simply not be cached.

 Consider using MEMORY_OR_DISK persistence.

 This can also happen if blocks were lost due to node failure.

 On Wed, Feb 18, 2015 at 3:19 PM, shahab shahab.mok...@gmail.com wrote:
  Hi,
 
  I have a cached RDD (I can see in UI that it is cached), but when I use
 this
  RDD , I can see that the RDD is partially recomputed (computed) again.
 It is
  partially because I can see in UI that some task are skipped (have a
 look
  at the attached figure).
 
  Now the question is 1: what causes a cached RDD to be recomputed again?
 and
  why somes tasks are skipped and some not??
 
  best,
  /Shahab
 
 
 
  -
  To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
  For additional commands, e-mail: user-h...@spark.apache.org

Re: Why cached RDD is recomputed again?

[Sean Owen]

The mostly likely explanation is that you wanted to put all the
partitions in memory and they don't all fit. Unless you asked to
persist to memory or disk, some partitions will simply not be cached.

Consider using MEMORY_OR_DISK persistence.

This can also happen if blocks were lost due to node failure.

On Wed, Feb 18, 2015 at 3:19 PM, shahab shahab.mok...@gmail.com wrote:
 Hi,

 I have a cached RDD (I can see in UI that it is cached), but when I use this
 RDD , I can see that the RDD is partially recomputed (computed) again. It is
 partially because I can see in UI that some task are skipped (have a look
 at the attached figure).

 Now the question is 1: what causes a cached RDD to be recomputed again? and
 why somes tasks are skipped and some not??

 best,
 /Shahab



 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Cached RDD

[Rishi Yadav]

Without caching, each action is recomputed. So assuming rdd2 and rdd3
result in separate actions answer is yes.

On Mon, Dec 29, 2014 at 7:53 PM, Corey Nolet cjno...@gmail.com wrote:

 If I have 2 RDDs which depend on the same RDD like the following:

 val rdd1 = ...

 val rdd2 = rdd1.groupBy()...

 val rdd3 = rdd1.groupBy()...


 If I don't cache rdd1, will it's lineage be calculated twice (one for rdd2
 and one for rdd3)?

Cached RDD

[Corey Nolet]

If I have 2 RDDs which depend on the same RDD like the following:

val rdd1 = ...

val rdd2 = rdd1.groupBy()...

val rdd3 = rdd1.groupBy()...


If I don't cache rdd1, will it's lineage be calculated twice (one for rdd2
and one for rdd3)?

Re: Cached RDD Block Size - Uneven Distribution

[anthonyjschu...@gmail.com]

I am having a similar problem:

I have a large dataset in HDFS and (for a few possible reason including a
filter operation, and some of my computation nodes simply not being hdfs
datanodes) have a large skew on my RDD blocks: the master node always has
the most, while the worker nodes have few... (and the non-hdfs nodes have
none)

What is the preferred way to rebalance this RDD across the cluster? Some of
my nodes are very underutilized :( I have tried:

.coalesce(15000, shuffle = false)

which helps a little, but things are still not evenly distributed...



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Cached-RDD-Block-Size-Uneven-Distribution-tp11286p12055.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Cached RDD Block Size - Uneven Distribution

[Patrick Wendell]

Are you directly caching files from Hadoop or are you doing some
transformation on them first? If you are doing a groupBy or some type of
transformation, then you could be causing data skew that way.


On Sun, Aug 3, 2014 at 1:19 PM, iramaraju iramar...@gmail.com wrote:

 I am running spark 1.0.0, Tachyon 0.5 and Hadoop 1.0.4.

 I am selecting a subset of a large dataset and trying to run queries on the
 cached schema RDD. Strangely, in web UI, I see the following.

 150 Partitions

 Block Name  Storage Level   Size in Memory ▴Size on Disk
  Executors
 rdd_30_68   Memory Deserialized 1x Replicated   307.5 MB
  0.0 B
 ip-172-31-45-100.ec2.internal:37796
 rdd_30_133  Memory Deserialized 1x Replicated   216.0 MB
  0.0 B
 ip-172-31-45-101.ec2.internal:55947
 rdd_30_18   Memory Deserialized 1x Replicated   194.2 MB
  0.0 B
 ip-172-31-42-159.ec2.internal:43543
 rdd_30_24   Memory Deserialized 1x Replicated   173.3 MB
  0.0 B
 ip-172-31-45-101.ec2.internal:55947
 rdd_30_70   Memory Deserialized 1x Replicated   168.2 MB
  0.0 B
 ip-172-31-18-220.ec2.internal:39847
 rdd_30_105  Memory Deserialized 1x Replicated   154.1 MB
  0.0 B
 ip-172-31-45-102.ec2.internal:36700
 rdd_30_79   Memory Deserialized 1x Replicated   153.9 MB
  0.0 B
 ip-172-31-45-99.ec2.internal:59538
 rdd_30_60   Memory Deserialized 1x Replicated   4.2 MB  0.0 B
 ip-172-31-45-102.ec2.internal:36700
 rdd_30_99   Memory Deserialized 1x Replicated   112.0 B 0.0 B
 ip-172-31-45-102.ec2.internal:36700
 rdd_30_90   Memory Deserialized 1x Replicated   112.0 B 0.0 B
 ip-172-31-45-102.ec2.internal:36700
 rdd_30_9Memory Deserialized 1x Replicated   112.0 B 0.0 B
 ip-172-31-18-220.ec2.internal:39847
 rdd_30_89   Memory Deserialized 1x Replicated   112.0 B 0.0 B
 ip-172-31-45-102.ec2.internal:36700

 What is strange to me is the size in Memory is mostly 112Bytes except for 8
 of them. ( I have 9 data files in Hadoop, which are well distributed 64mb
 blocks ).

 The tasks processing the rdd are getting stuck after finishing few initial
 tasks. I am wondering, it is because, the spark has hit the large blocks
 and
 trying to process them on one worker per task.

 Any suggestions on how I can distribute them more evenly (Size of blocks) ?
 And why my hadoop blocks are nicely even and spark cached RDD has such a
 uneven distribution ? Any help is appreciated.

 Regards
 Ram



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/Cached-RDD-Block-Size-Uneven-Distribution-tp11286.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

Cached RDD Block Size - Uneven Distribution

[iramaraju]

I am running spark 1.0.0, Tachyon 0.5 and Hadoop 1.0.4.

I am selecting a subset of a large dataset and trying to run queries on the
cached schema RDD. Strangely, in web UI, I see the following.

150 Partitions

Block Name  Storage Level   Size in Memory ▴Size on Disk
Executors
rdd_30_68   Memory Deserialized 1x Replicated   307.5 MB0.0 B
ip-172-31-45-100.ec2.internal:37796
rdd_30_133  Memory Deserialized 1x Replicated   216.0 MB0.0 B
ip-172-31-45-101.ec2.internal:55947
rdd_30_18   Memory Deserialized 1x Replicated   194.2 MB0.0 B
ip-172-31-42-159.ec2.internal:43543
rdd_30_24   Memory Deserialized 1x Replicated   173.3 MB0.0 B
ip-172-31-45-101.ec2.internal:55947
rdd_30_70   Memory Deserialized 1x Replicated   168.2 MB0.0 B
ip-172-31-18-220.ec2.internal:39847
rdd_30_105  Memory Deserialized 1x Replicated   154.1 MB0.0 B
ip-172-31-45-102.ec2.internal:36700
rdd_30_79   Memory Deserialized 1x Replicated   153.9 MB0.0 B
ip-172-31-45-99.ec2.internal:59538
rdd_30_60   Memory Deserialized 1x Replicated   4.2 MB  0.0 B
ip-172-31-45-102.ec2.internal:36700
rdd_30_99   Memory Deserialized 1x Replicated   112.0 B 0.0 B
ip-172-31-45-102.ec2.internal:36700
rdd_30_90   Memory Deserialized 1x Replicated   112.0 B 0.0 B
ip-172-31-45-102.ec2.internal:36700
rdd_30_9Memory Deserialized 1x Replicated   112.0 B 0.0 B
ip-172-31-18-220.ec2.internal:39847
rdd_30_89   Memory Deserialized 1x Replicated   112.0 B 0.0 B
ip-172-31-45-102.ec2.internal:36700

What is strange to me is the size in Memory is mostly 112Bytes except for 8
of them. ( I have 9 data files in Hadoop, which are well distributed 64mb
blocks ).

The tasks processing the rdd are getting stuck after finishing few initial
tasks. I am wondering, it is because, the spark has hit the large blocks and
trying to process them on one worker per task.

Any suggestions on how I can distribute them more evenly (Size of blocks) ?
And why my hadoop blocks are nicely even and spark cached RDD has such a
uneven distribution ? Any help is appreciated.

Regards
Ram



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Cached-RDD-Block-Size-Uneven-Distribution-tp11286.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org