Disclaimer: I haven't done this myself, or even attempted to compile the code
below, but in hopes it makes your life a little easier...
Here's some Erlang that should allow you to abstract away the checks. Assuming
that you have a function named "real_map_fun" that takes a single Riak object
as an argument, you should be able to reuse "check_found" and "check_tombstone"
below and wrap the call to "real_map_fun" in a function like "map_something".
map_something(Obj, _, _) ->
check_found(Obj, fun real_map_fun/1).
check_found({error, notfound}, _Fun) ->
[];
check_found(Obj, Fun) ->
check_tombstone(dict:is_key(<<"X-Riak-Deleted">>,
riak_object:get_metadata(Obj)),
Obj, Fun).
check_tombstone(true, _Obj, _Fun) ->
[];
check_tombstone(false, Obj, Fun) ->
Fun(Obj).
-John
On Mar 6, 2013, at 4:45 PM, Jeremy Raymond <[email protected]> wrote:
> Thanks for the response. To handle not founds and tombstones I need this in
> every map function?
>
> map_something({error, notfound}, _, _) ->
> [];
> map_something(RiakObj, _, _) ->
> Metadata = riak_object:get_metadata(RiakObj),
> case dict:is_key(<<"X-Riak-Deleted">>, Metadata) of
> true ->
> []; % I have a tombstone
> false ->
> [ok] % I have a valid item
> end.
>
> Is there a better or built-in way to do this filtering?
>
> --
> Jeremy
>
>
> On Wed, Mar 6, 2013 at 4:08 PM, John Daily <[email protected]> wrote:
> I'm sorry this went unanswered, Jeremy. Thanks for the follow up.
>
> Your code needs to be able to handle notfound errors and tombstones[1]
> regardless of ownership handoff. The coverage for the 2i or listkeys input
> will be calculated up front, with the work distributed to a node where the
> key is expected to be found, but it's always possible that the node selected
> may not have the data you want due to network or system errors that haven't
> yet healed.
>
> Both the notfound and the fitting error in the logs (which is benign, and
> related to the notfound problem) should be less common under Riak 1.3,
> although ownership handoff will still exacerbate the problem.
>
> [1] https://github.com/basho/riak_kv/issues/358
>
> -John Daily
> Technical Evangelist
> [email protected]
>
>
> On Mar 6, 2013, at 3:14 PM, Jeremy Raymond <[email protected]> wrote:
>
>> So I should expect {error, notfound} inputs to map jobs while ownership
>> handoff is in progress? Are the not found items actually unavailable during
>> handoff or is this just not found on the old node, but will be picked up by
>> the new node during the same mapreduce job?
>>
>> --
>> Jeremy
>>
>>
>> On Thu, Feb 28, 2013 at 11:28 AM, Jeremy Raymond <[email protected]> wrote:
>> Yesterday I added a new node to my cluster. During the time when ownership
>> handoff was happening (several hours of work) mapreduce map functions were
>> receiving {error, notfound} as inputs. My Erlang mapred functions weren't
>> designed to handle this. They hadn't encountered this before during normal
>> operation. After the ownership handoff process completed the {error,
>> notfound} inputs have stopped.
>>
>> Any explanations for the {error, notfound} inputs during ownership handoff?
>> Is this because a node is attempting to process an object now moved to
>> another node? If this is the case would the notfound object be found on the
>> other node in the same mapreduce job (i.e. still visible to the overall
>> mapred process)? Should I assume {error, notfound} inputs for all mapred
>> jobs as a valid possible input and always handle it?
>>
>> I've also accumulated about 50MB of "Pipe worker startup failed:fitting was
>> gone before startup" on each node during the ownership_transfer process.
>> These messages are benign?
>>
>> Thanks a lot.
>>
>> --
>> Jeremy
>>
>> _______________________________________________
>> riak-users mailing list
>> [email protected]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
