[
https://issues.apache.org/jira/browse/SOLR-9905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Bernstein updated SOLR-9905:
---------------------------------
Fix Version/s: 6.4
master (7.0)
> Add NullStream to isolate the performance of the ExportWriter
> -------------------------------------------------------------
>
> Key: SOLR-9905
> URL: https://issues.apache.org/jira/browse/SOLR-9905
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Joel Bernstein
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9905.patch
>
>
> The NullStream is a utility function to test the raw performance of the
> ExportWriter. This is a nice utility to have to diagnose bottlenecks in
> streaming MapReduce operations. The NullStream will allow developers to test
> the performance of the shuffling (Sorting, Partitioning, Exporting) in
> isolation from the reduce operation (Rollup, Join, Group, etc..).
> The NullStream simply iterates it's internal stream and eats the tuples. It
> returns a single Tuple from each worker with the number of Tuples processed.
> The idea is to iterate the stream without additional overhead so the
> performance of the underlying stream can be isolated.
> Sample syntax:
> {code}
> parallel(collection2, workers=7, sort="nullCount desc",
> null(search(collection1,
> q=*:*,
> fl="id",
> sort="id desc",
> qt="/export",
> wt="javabin",
> partitionKeys=id)))
> {code}
> In the example above the NullStream is sent to 7 workers. Each worker will
> iterate the search() expression and the NullStream will eat the tuples so the
> raw performance of the search() can be understood.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
