cryptoe commented on code in PR #13274:
URL: https://github.com/apache/druid/pull/13274#discussion_r1012907059
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerMemoryParameters.java:
##########
@@ -272,23 +347,78 @@ static int computeMaxWorkers(
final long bundleMemory = memoryPerBundle(maxMemoryInJvm, numWorkersInJvm,
numProcessingThreadsInJvm);
// Inverse of memoryNeededForInputChannels.
- return Ints.checkedCast((bundleMemory - PROCESSING_MINIMUM_BYTES) /
STANDARD_FRAME_SIZE - 1);
+ return Math.max(0, Ints.checkedCast((bundleMemory -
PROCESSING_MINIMUM_BYTES) / STANDARD_FRAME_SIZE - 1));
}
+ /**
+ * Maximum number of workers that may exist in the current JVM.
+ */
+ private static int computeNumWorkersInJvm(final Injector injector)
+ {
+ final AppenderatorsManager appenderatorsManager =
injector.getInstance(AppenderatorsManager.class);
+
+ if (appenderatorsManager instanceof UnifiedIndexerAppenderatorsManager) {
+ // CliIndexer
+ return injector.getInstance(WorkerConfig.class).getCapacity();
+ } else {
+ // CliPeon
+ return 1;
+ }
+ }
+
+ /**
+ * Maximum number of concurrent processors that exist in the current JVM.
+ */
+ private static int computeNumProcessorsInJvm(final Injector injector)
+ {
+ return injector.getInstance(Bouncer.class).getMaxCount();
+ }
+
+ /**
+ * Compute the memory allocated to each worker. Includes anything that
exists outside of processing bundles.
+ *
+ * Today, we only look at one thing: the amount of memory taken up by
+ * {@link org.apache.druid.msq.statistics.ClusterByStatisticsCollector}.
This is the single largest source of memory
+ * usage outside processing bundles.
+ */
+ private static long memoryPerWorker(
+ final long maxMemoryInJvm,
+ final int numWorkersInJvm
+ )
+ {
+ final long usableMemory = (long) (maxMemoryInJvm * USABLE_MEMORY_FRACTION);
+ final long memoryForWorkers = (long) Math.min(
Review Comment:
Nice. This is really cool and will go a long way in fixing the memory issues
of sketches.
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerStageTracker.java:
##########
@@ -216,6 +219,10 @@ void start()
*/
void finish()
{
+ if (resultKeyStatisticsCollector != null) {
+ resultKeyStatisticsCollector.clear();
Review Comment:
Nice catch !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
