[ 
https://issues.apache.org/jira/browse/SOLR-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209557#comment-17209557
 ] 

Munendra S N commented on SOLR-14470:
-------------------------------------

[~ab]
I was looking into an intermittent test failure in TestExportWriter(from 
badApple report) but it wasn't failing on master even with beasting but it 
failed on 8x branch. On comparing the diff, it seems the commit 
{{a5543dfb5112d12b9616f2d09cd07e9805b177de}} is not backported to 8x. I locally 
backported to 8x and did beasting(50 iterations) and there were no test 
failures. 
Just checking with you before backporting test fixes

> Add streaming expressions to /export handler
> --------------------------------------------
>
>                 Key: SOLR-14470
>                 URL: https://issues.apache.org/jira/browse/SOLR-14470
>             Project: Solr
>          Issue Type: Improvement
>          Components: Export Writer, streaming expressions
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>             Fix For: 8.6
>
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Many streaming scenarios would greatly benefit from the ability to perform 
> partial rollups (or other transformations) as early as possible, in order to 
> minimize the amount of data that has to be sent from shards to the 
> aggregating node.
> This can be implemented as a subset of streaming expressions that process the 
> data directly inside each local {{ExportHandler}} and outputs only the 
> records from the resulting stream. 
> Conceptually it would be similar to the way Hadoop {{Combiner}} works. As is 
> the case with {{Combiner}}, because the input data is processed in batches 
> there would be no guarantee that only 1 record per unique sort values would 
> be emitted - in fact, in most cases multiple partial aggregations would be 
> emitted. Still, in many scenarios this would allow reducing the amount of 
> data to be sent by several orders of magnitude.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to