[
https://issues.apache.org/jira/browse/SOLR-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878869#comment-17878869
]
Joel Bernstein edited comment on SOLR-17430 at 9/3/24 12:43 PM:
----------------------------------------------------------------
One thought on the design I have is to use a blocking queue with a global
thread pool of output threads waiting on the queue. In this situation the
output/writer threads block indefinitely, and handle output writing for all
export queries. The reader thread, which is the main execution thread for the
query, simply pushes batches of materialized docs to the queue.
was (Author: joel.bernstein):
One thought on the design I have is to use a blocking queue with a global
thread pool of output threads waiting on the queue. In this situation the
output/writer threads block indefinitely. The reader thread, which is the main
execution thread for the query, simply pushes batches of materialized docs to
the queue.
> Redesign ExportWriter / ExportBuffers to work better with large batchSizes
> and slow consumption
> -----------------------------------------------------------------------------------------------
>
> Key: SOLR-17430
> URL: https://issues.apache.org/jira/browse/SOLR-17430
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Chris M. Hostetter
> Priority: Major
>
> As mentioned in SOLR-17416, the design of the {{ExportBuffers}} class used by
> the {{ExportHandler}} is brittle and the absolutely time limit on how long
> the buffer swapping threads will wait for eachother isn't suitable for very
> long running streaming expressions...
> {quote}The problem however is that this 600 second timeout may not be enough
> to account for really slow downstream consumption of the data. With really
> large collections, and really complicated streaming expressions, this can
> happen even when well behaved clients that are actively trying to consume
> data.
> {quote}
> ...but another sub-optimal aspect of this buffer swapping design is that the
> "writer" thread is initially completely blocked, and can't write out a single
> document, until the "filler" thread has read the full {{batchSize}} of
> documents into it's buffer and opted to swap. Likewise, after buffer
> swapping has occured at least once, any document in the {{outputBuffer}} that
> the writer has already processed hangs around, taking up ram, until the next
> swap, while one of the threads is idle. If {{{}batchSize=30000{}}}, and the
> "filler" thread is ready to go with a full {{fillBuffer}} while the "writer"
> has only been able to emit 29999 of the documents in it's {{outputBuffer}}
> documents before being blocked and forced to wait (due to the downstream
> consumer of the output bytes) before it can emit the last document in it's
> batch – that means both the "writer" thread and the "filler" thread are
> stalled, taking up 2x the batchSize of ram, even though half of that is data
> that is no longer needed.
> The bigger the {{batchSize}} the worse the initial delay (and steady state
> wasted RAM) is.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
