Tom, 
    Are you suggesting two RDDs, one with loss and another for the rest info, 
using zip to tie them together, but do update on loss RDD (copy) ?

Chester

Sent from my iPhone

On Apr 28, 2014, at 9:45 AM, Tom Vacek <[email protected]> wrote:

> I'm not sure what I said came through.  RDD zip is not hacky at all, as it 
> only depends on a user not changing the partitioning.  Basically, you would 
> keep your losses as an RDD[Double] and zip whose with the RDD of examples, 
> and update the losses.  You're doing a copy (and GC) on the RDD of losses 
> each time, but this is negligible.
> 
> 
> On Mon, Apr 28, 2014 at 11:33 AM, Sung Hwan Chung <[email protected]> 
> wrote:
>> Yes, this is what we've done as of now (if you read earlier threads). And we 
>> were saying that we'd prefer if Spark supported persistent worker memory 
>> management in a little bit less hacky way ;)
>> 
>> 
>> On Mon, Apr 28, 2014 at 8:44 AM, Ian O'Connell <[email protected]> wrote:
>>> A mutable map in an object should do what your looking for then I believe. 
>>> You just reference the object as an object in your closure so it won't be 
>>> swept up when your closure is serialized and you can reference variables of 
>>> the object on the remote host then. e.g.:
>>> 
>>> object MyObject {
>>>   val mmap = scala.collection.mutable.Map[Long, Long]()
>>> }
>>> 
>>> rdd.map { ele =>
>>> MyObject.mmap.getOrElseUpdate(ele, 1L)
>>> ...
>>> }.map {ele =>
>>> require(MyObject.mmap(ele) == 1L)
>>> 
>>> }.count
>>> 
>>> Along with the data loss just be careful with thread safety and multiple 
>>> threads/partitions on one host so the map should be viewed as shared 
>>> amongst a larger space. 
>>> 
>>> 
>>> 
>>> Also with your exact description it sounds like your data should be encoded 
>>> into the RDD if its per-record/per-row:  RDD[(MyBaseData, 
>>> LastIterationSideValues)] 
>>> 
>>> 
>>> 
>>> On Mon, Apr 28, 2014 at 1:51 AM, Sung Hwan Chung <[email protected]> 
>>> wrote:
>>>> In our case, we'd like to keep memory content from one iteration to the 
>>>> next, and not just during a single mapPartition call because then we can 
>>>> do more efficient computations using the values from the previous 
>>>> iteration.
>>>> 
>>>> So essentially, we need to declare objects outside the scope of the 
>>>> map/reduce calls (but residing in individual workers), then those can be 
>>>> accessed from the map/reduce calls.
>>>> 
>>>> We'd be making some assumptions as you said, such as - RDD partition is 
>>>> statically located and can't move from worker to another worker unless the 
>>>> worker crashes.
>>>> 
>>>> 
>>>> 
>>>> On Mon, Apr 28, 2014 at 1:35 AM, Sean Owen <[email protected]> wrote:
>>>>> On Mon, Apr 28, 2014 at 9:30 AM, Sung Hwan Chung 
>>>>> <[email protected]> wrote:
>>>>>> Actually, I do not know how to do something like this or whether this is 
>>>>>> possible - thus my suggestive statement.
>>>>>> 
>>>>>> Can you already declare persistent memory objects per worker? I tried 
>>>>>> something like constructing a singleton object within map functions, but 
>>>>>> that didn't work as it seemed to actually serialize singletons and pass 
>>>>>> it back and forth in a weird manner.
>>>>> 
>>>>> Does it need to be persistent across operations, or just persist for the 
>>>>> lifetime of processing of one partition in one mapPartition? The latter 
>>>>> is quite easy and might give most of the speedup.
>>>>> 
>>>>> Maybe that's 'enough', even if it means you re-cache values several times 
>>>>> in a repeated iterative computation. It would certainly avoid managing a 
>>>>> lot of complexity in trying to keep that state alive remotely across 
>>>>> operations. I'd also be interested if there is any reliable way to do 
>>>>> that, though it seems hard since it means you embed assumptions about 
>>>>> where particular data is going to be processed.
> 

Reply via email to