Attached is the screen shot of the storage tab details for the cached rdd. The host highlighted and at the end of the list is driver machine.
[cid:image001.png@01D1496C.FD79AB40] -regards Seemanto Barua From: Barua, Seemanto (US) Sent: Thursday, January 07, 2016 12:43 PM To: 'premsure...@gmail.com' Cc: 'user@spark.apache.org' Subject: Re: Question in rdd caching in memory using persist I have a standalone cluster. spark version is 1.3.1 From: Prem Sure [mailto:premsure...@gmail.com] Sent: Thursday, January 07, 2016 12:32 PM To: Barua, Seemanto (US) Cc: spark users <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Question in rdd caching in memory using persist are you running standalone - local mode or cluster mode. executor and driver existance differ based on setup type. snapshot of your env UI would be helpful to say On Thu, Jan 7, 2016 at 11:51 AM, <seemanto.ba...@nomura.com<mailto:seemanto.ba...@nomura.com>> wrote: Hi, After I called rdd.persist(MEMORY_ONLY_SER), I see the driver listed as one of the ‘executors’ participating in holding the partitions of the rdd in memory, the memory usage shown against the driver is 0. This I see in the storage tab of the spark ui. Why is the driver shown on the ui ? Will it ever hold rdd partitions when caching. -regards Seemanto Barua PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation. PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
