imply-jhan commented on a change in pull request #11536:
URL: https://github.com/apache/druid/pull/11536#discussion_r703832810
##########
File path: docs/configuration/index.md
##########
@@ -1343,7 +1343,8 @@ Additional peon configs include:
|`druid.peon.mode`|Choices are "local" and "remote". Setting this to local
means you intend to run the peon as a standalone process (Not
recommended).|remote|
|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/task`|
-|`druid.indexer.task.useLegacyBatchProcessing`|If false, native batch
ingestion will use a new, recommended, code path with memory optimized code for
the segment creation phase. If true it will use the previous code path for the
create segments phase of batch ingestion. This does not apply to streaming
ingestion, just to batch. This setting should only be used when a bug is
suspected or found in the new optimized batch ingestion code. If a bug is
suspected or found, you can set this flag to `true` to fall back to previous,
working but more memory intensive, code path.|`false`|
+|`druid.indexer.task.batchMemoryMappedIndex`|DEPRECATED: Use
`druid.indexer.task.batchProcessingMode` instead. If false, native batch
ingestion will use a new, recommended, code path with memory optimized code for
the segment creation phase. If true it will use the previous code path for the
create segments phase of batch ingestion. This does not apply to streaming
ingestion, just to batch. This setting should only be used when a bug is
suspected or found in the new optimized batch ingestion code. If a bug is
suspected or found, you can set this flag to `true` to fall back to previous,
working but more memory intensive, code path.|`false`|
Review comment:
Maybe call out that set batchMemoryMappedIndex to true will set
batchProcessingMode to LEGACY and overwrite the batchProcessingMode value.
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
##########
@@ -117,7 +131,19 @@ public TaskConfig(
this.shuffleDataLocations = shuffleDataLocations;
}
this.ignoreTimestampSpecForDruidInputSource =
ignoreTimestampSpecForDruidInputSource;
- this.useLegacyBatchProcessing = useLegacyBatchProcessing;
+
+ this.batchMemoryMappedIndex = batchMemoryMappedIndex;
+ // Conflict resolution. Assume that if batchMemoryMappedIndex is set
(since false is the default) that
+ // the user changed it intentionally to use legacy, in this case oveeride
batchProcessingMode and also
+ // set it to legacy else just use batchProcessingMode and don't pay
attention to batchMemoryMappedIndexMode:
+ if (batchMemoryMappedIndex) {
+ this.batchProcessingMode = BatchProcesingMode.LEGACY;
+ } else if (EnumUtils.isValidEnum(BatchProcesingMode.class,
batchProcesingMode)) {
+ this.batchProcessingMode =
BatchProcesingMode.valueOf(batchProcesingMode);
+ } else {
+ // batchProcessingMode input string is invalid, just use the
default...log message somewhere???
+ this.batchProcessingMode = BatchProcesingMode.CLOSED_SEGMENTS; // Default
+ }
Review comment:
Can we log the batchProcessingMode value after this if block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
