drcrallen opened a new pull request #6014: Optionally refuse to consume new data until the prior chunk is being consumed URL: https://github.com/apache/incubator-druid/pull/6014 This PR adds in an optional boolean query context field called `enableBrokerBackpressure`. The impact of when this flag is set to `true` is to force the `DirectDruidClient` to stop collecting new data until this result group is actively being consumed. In the prior implementation, parallel query results across the cluster will be collected in a `LinkedBlockingQueue` without bound. This means that ALL results are potentially accumulated as channel streams without the ability to limit the outstanding data that has not been consumed. This coupled with a highly single threaded folding operation (see https://github.com/apache/incubator-druid/pull/5913) on result merging means that it can be an unknown length of time until the results are even attempted to be consumed. The intention of this change is that the result that is actively being folded into a sequence has one channel input stream that is being consumed to be folded, and one in transit in parallel. This is enforced by using the `transfer` method instead of the `put` method to add new data to the enumerator on the output side. This feature is feature-flagged by a context field so should have no impact on anyone who does not enable this feature. Work before merging: - [ ] Test on an actual cluster
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
