Hi,
The main question: how would I use riak pipes in a mapred request via
riak-erlang-client?
Long version:
I have a dataset of about 20K items (N). One of the main operations is
to produce K different views on each element and consolidate the
resulting views. However each of those K views are very much
independent of each other, in other words it should work somehow like
this:
_ View 1 -> output 1
/
N items - ...
\
- View K -> output K
Each of the items might or might not be represented in a particular
view.
The straightforward approach is to use a Map -> Reduce where
Map generates corresponding representation of the item in relevant
views
Reduce basically consolidates data for each view
What feels wrong is instead of K reduce simple functions (consolidate
the data) there would be one a bit more complex that would have to split
input data into streams and then combine consolidated views into a
structure to pass to the next iteration of reduce.
Riak pipes seem to be a bit more appropriate for the use case, however
it's not really clear if it's possible to use them via riak pb client
interface. And, if direct usage is not really possible, I wonder if
riak pipes still could be used to somehow optimise the data processing.
--
Misha
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
