clintropolis commented on code in PR #13803:
URL: https://github.com/apache/druid/pull/13803#discussion_r1146910014
##########
processing/src/test/java/org/apache/druid/segment/IndexBuilder.java:
##########
@@ -267,4 +395,44 @@ private static IncrementalIndex
buildIncrementalIndexWithRows(
}
return incrementalIndex;
}
+
+ private static IncrementalIndex buildIncrementalIndexWithInputSource(
+ IncrementalIndexSchema schema,
+ InputSource inputSource,
+ InputFormat inputFormat,
+ @Nullable TransformSpec transformSpec,
+ File inputSourceTmpDir,
+ int maxRows
+ )
+ {
+ Preconditions.checkNotNull(schema, "schema");
+ Preconditions.checkNotNull(inputSource, "inputSource");
+ Preconditions.checkNotNull(inputFormat, "inputFormat");
+ Preconditions.checkNotNull(inputSourceTmpDir, "inputSourceTmpDir");
+
+ final IncrementalIndex incrementalIndex = new
OnheapIncrementalIndex.Builder()
+ .setIndexSchema(schema)
+ .setMaxRowCount(maxRows)
+ .build();
+ TransformSpec tranformer = transformSpec != null ? transformSpec :
TransformSpec.NONE;
+ InputRowSchema rowSchema = new InputRowSchema(schema.getTimestampSpec(),
schema.getDimensionsSpec(), null);
+ InputSourceReader reader = inputSource.reader(rowSchema, inputFormat,
inputSourceTmpDir);
+ InputSourceReader transformingReader = tranformer.decorate(reader);
+ try (CloseableIterator<InputRow> rowIterator = transformingReader.read()) {
+ while (rowIterator.hasNext()) {
+ incrementalIndex.add(rowIterator.next());
Review Comment:
Hmm, good question, this looks wired up to the `maxRowCount` but callers are
not checking `canAppendRow` or anything like that and `buildIncrementalIndex`
can only return a single `IncrementalIndex` so I think all it can do is
explode. There is also `intermediatePersistSize` which can be used to force a
bunch of incremental indexes to be written when `mergeIndexes` is called to
make sure that segment merging happens.
Nothing much seems to be explicitly setting either of these things, and
maybe could be removed or reworked in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
