Re: How executor Understand which RDDs needed to be persist from the submitted Task

Thanks for your help! Iacovos On 1/9/20 5:49 PM, Wenchen Fan wrote: You can take a look at ShuffleMapTask.runTask. It's not just a function. On Thu, Jan 9, 2020 at 11:25 PM Jack Kolokasis <mailto:koloka...@ics.forth.gr>> wrote: Thanks for the help. I read that driver

Re: How executor Understand which RDDs needed to be persist from the submitted Task

and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed. On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <mailto:koloka...@ics.forth.gr>> wrote: Hello all, I want to find when a Task that is sended

How executor Understand which RDDs needed to be persist from the submitted Task

Hello all, I want to find when a Task that is sended by Driver to executor contains a call to function persist(). I am trying to read the submitted function that driver send to executor but I could not find any call to persist() method. Do you know how executor understand which RDDs needed to

Re: Tungsten Memory Consumer

explain me why this happens ? HeapMemoryAllocator would not always called by MemoryConsumer  ? --Iacovos On 11/02/2019 11:06 πμ, Wenchen Fan wrote: what do you mean by ''Tungsten Consumer"? On Fri, Feb 8, 2019 at 6:11 PM Jack Kolokasis <mailto:koloka...@ics.forth.gr>> wrote:

Tungsten Memory Consumer

looking forward for your reply. --Jack Kolokasis - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

TaskMemoryManager

application execution where TaskMemoryManager does not need to allocate extra memory. Can anyone explain to me why this happens. Thanks a lot and I am looking forward for your reply. --Jack Kolokasis - To unsubscribe e-mail: dev

Profile Spark Executors

Hi all,     I try to profile my spark executors performance when use on Heap persistent level in compare to use off-Heap persistent level. I use statsd-jvm-profiler to profile each executor. From the results i see that application spends 71,92% of its threads running the method

Maven

Hello,    is there any way to use my local custom - Spark as dependency while I am using maven to compile my applications ? Thanks for your reply, --Iacovos - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: welcome a new batch of committers

Congratulations to all !! -Iacovos On 03/10/2018 12:54 μμ, Ted Yu wrote: Congratulations to all ! Original message From: Jungtaek Lim Date: 10/3/18 2:41 AM (GMT-08:00) To: Marco Gaido Cc: dev Subject: Re: welcome a new batch of committers Congrats all! You all deserved

Off Heap Memory

Hello,     I recently start studying the Spark's memory management system. More spesifically I want to understand how spark use the off-Heap memory. Interanlly I saw, that there are two types of offHeap memory. (offHeapExecutionMemoryPool and offHeapStorageMemoryPool).     How Spark use the

Off Heap Memory

Hello,     I recently start studying the Spark's memory management system. My question is about the offHeapExecutionMemoryPool and offHeapStorageMemoryPool.     1. How Spark use the offHeapExecutionMemoryPool ?     2. How use the offHeap memory (I understand the allocation side), but it is