cryptoe commented on code in PR #14061:
URL: https://github.com/apache/druid/pull/14061#discussion_r1162300480
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:
##########
@@ -588,21 +588,20 @@ private QueryDefinition initializeQueryDefAndState(final
Closer closer)
.orElse(MSQWarnings.DEFAULT_MAX_PARSE_EXCEPTIONS_ALLOWED);
}
+ WorkerStorageParameters workerStorageParameters =
+
WorkerStorageParameters.createProductionInstanceForController(context.injector(),
isDurableStorageEnabled);
+
ImmutableMap.Builder<String, Object> taskContextOverridesBuilder =
ImmutableMap.builder();
taskContextOverridesBuilder
.put(
MultiStageQueryContext.CTX_DURABLE_SHUFFLE_STORAGE,
isDurableStorageEnabled
).put(
MultiStageQueryContext.CTX_COMPOSED_INTERMEDIATE_SUPER_SORTER_STORAGE,
Review Comment:
Should we remove this parameter ?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java:
##########
@@ -78,6 +78,7 @@
private static final Period DEFAULT_DIRECTORY_LOCK_TIMEOUT = new
Period("PT10M");
private static final Period DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT = new
Period("PT5M");
private static final boolean DEFAULT_STORE_EMPTY_COLUMNS = true;
+ private static final long DEFAULT_TMP_STORAGE_BYTES_PER_TASK =
Long.MAX_VALUE; // TODO: check!!!
Review Comment:
Is this a left over todo ?
Where is the property `tmpStorageBytesPerTask` documented ?
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:
##########
@@ -588,21 +588,20 @@ private QueryDefinition initializeQueryDefAndState(final
Closer closer)
.orElse(MSQWarnings.DEFAULT_MAX_PARSE_EXCEPTIONS_ALLOWED);
}
+ WorkerStorageParameters workerStorageParameters =
+
WorkerStorageParameters.createProductionInstanceForController(context.injector(),
isDurableStorageEnabled);
+
ImmutableMap.Builder<String, Object> taskContextOverridesBuilder =
ImmutableMap.builder();
taskContextOverridesBuilder
.put(
MultiStageQueryContext.CTX_DURABLE_SHUFFLE_STORAGE,
isDurableStorageEnabled
).put(
MultiStageQueryContext.CTX_COMPOSED_INTERMEDIATE_SUPER_SORTER_STORAGE,
-
MultiStageQueryContext.isComposedIntermediateSuperSorterStorageEnabled(
- task.getQuerySpec().getQuery().context()
- )
+ isDurableStorageEnabled // If Durable Storage is enabled, then
super sorter intermediate storage can be enabled.
).put(
MultiStageQueryContext.CTX_INTERMEDIATE_SUPER_SORTER_STORAGE_MAX_LOCAL_BYTES,
Review Comment:
Lets update the documentation in `MultiStageQueryContext` for these params.
Also the user facing docs for durableStorage would need to be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
