[
https://issues.apache.org/jira/browse/SOLR-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716486#comment-13716486
]
Andrzej Bialecki commented on SOLR-5069:
-----------------------------------------
Exciting idea! Almost as exciting as SolrCloud on MapReduce :)
A few comments:
# distributed map-reduce in reality is a sequence of:
## split input and assign splits to M nodes
## apply map() on M nodes in parallel
##* for large datasets the emitted data from mappers is spooled to disk
## "shuffle" - ie. partition and ship emitted data from M mappers into N
reducers
##* (wait until all mappers are done, so that each partition's key-space is
complete)
## sort by key in each of N reducers, collecting values for each key
##* again, for large datasets this is a disk-based sort
## apply N reducers in parallel and emit final output (in N parts)
# if I understand it correctly the model that you presented has some
limitations:
## as many input splits as there are shards (and consequently as many mappers)
## single reducer. Theoretically it should be possible to use N nodes to act as
reducers if you implement the concept of partitioner - this would cut down the
memory load on each reducer node. Of course, streaming back the results would
be a challenge, but saving them into a collection should work just fine.
## no "shuffling" - all data from mappers will go to a single reducer
## no intermediate storage of data, all intermediate values need to fit in
memory
## what about the sorting phase? I assume it's an implicit function in the
reducedMap (treemap?)
# since all fine-grained emitted values from map end up being sent to 1
reducer, which has to collect all this data in memory first before applying the
reduce() op, the concept of a map-side combiner seems useful, to be able to
quickly minimize the amount of data to be sent to reducer.
# it would be very easy to OOM your Solr nodes at the reduce phase. There
should be some built-in safety mechanism for this.
# what parts of Solr are available in the script's context? Making all Solr API
available could lead to unpredictable side-effects, so this set of APIs needs
to be curated. E.g. I think it would make sense to make analyzer factories
available.
And finally, an observation: regular distributed search can be viewed as a
special case of map-reduce computation ;)
> MapReduce for SolrCloud
> -----------------------
>
> Key: SOLR-5069
> URL: https://issues.apache.org/jira/browse/SOLR-5069
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
> Reporter: Noble Paul
> Assignee: Noble Paul
>
> Solr currently does not have a way to run long running computational tasks
> across the cluster. We can piggyback on the mapreduce paradigm so that users
> have smooth learning curve.
> * The mapreduce component will be written as a RequestHandler in Solr
> * Works only in SolrCloud mode. (No support for standalone mode)
> * Users can write MapReduce programs in Javascript or Java. First cut would
> be JS ( ? )
> h1. sample word count program
> h2.how to invoke?
> http://host:port/solr/mapreduce?map=<map-script>&reduce=<reduce-script>&sink=collectionX
> h3. params
> * map : A javascript implementation of the map program
> * reduce : a Javascript implementation of the reduce program
> * sink : The collection to which the output is written. If this is not passed
> , the request will wait till completion and respond with the output of the
> reduce program and will be emitted as a standard solr response. . If no sink
> is passed the request will be redirected to the "reduce node" where it will
> wait till the process is complete. If the sink param is passed ,the rsponse
> will contain an id of the run which can be used to query the status in
> another command.
> * reduceNode : Node name where the reduce is run . If not passed an arbitrary
> node is chosen
> The node which received the command would first identify one replica from
> each slice where the map program is executed . It will also identify one
> another node from the same collection where the reduce program is run. Each
> run is given an id and the details of the nodes participating in the run will
> be written to ZK (as an ephemeral node).
> h4. map script
> {code:JavaScript}
> var res = $.streamQuery(*:*);//this is not run across the cluster. //Only on
> this index
> while(res.hasMore()){
> var doc = res.next();
> var txt = doc.get(“txt”);//the field on which word count is performed
> var words = txt.split(" ");
> for(i = 0; i < words.length; i++){
> $.map(words[i],{‘count’:1});// this will send the map over to //the
> reduce host
> }
> }
> {code}
> Essentially two threads are created in the 'map' hosts . One for running the
> program and the other for co-ordinating with the 'reduce' host . The maps
> emitted are streamed live over an http connection to the reduce program
> h4. reduce script
> This script is run in one node . This node accepts http connections from map
> nodes and the 'maps' that are sent are collected in a queue which will be
> polled and fed into the reduce program. This also keeps the 'reduced' data in
> memory till the whole run is complete. It expects a "done" message from all
> 'map' nodes before it declares the tasks are complete. After reduce program
> is executed for all the input it proceeds to write out the result to the
> 'sink' collection or it is written straight out to the response.
> {code:JavaScript}
> var pair = $.nextMap();
> var reduced = $.getCtx().getReducedMap();// a hashmap
> var count = reduced.get(pair.key());
> if(count === null) {
> count = {“count”:0};
> reduced.put(pair.key(), count);
> }
> count.count += pair.val().count ;
> {code}
> TBD
> * The format in which the output is written to the target collection, I
> assume the reducedMap will have values mapping to the schema of the collection
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
