FrankChen021 commented on a change in pull request #10383:
URL: https://github.com/apache/druid/pull/10383#discussion_r521079741
##########
File path:
indexing-service/src/test/java/org/apache/druid/indexing/overlord/sampler/InputSourceSamplerTest.java
##########
@@ -1163,6 +1172,121 @@ public void testIndexParseException() throws IOException
);
}
+ /**
+ *
+ * This case tests sampling for multiple json lines in one text block
+ * Currently only RecordSupplierInputSource supports this kind of input, see
https://github.com/apache/druid/pull/10383 for more information
+ *
+ * This test combines illegal json block and legal json block together to
verify:
+ * 1. all lines in the illegal json block should not be parsed
+ * 2. the illegal json block should not affect the processing of the 2nd
record
+ * 3. all lines in legal json block should be parsed successfully
+ *
+ */
+ @Test
+ public void testMultipleJsonStringInOneBlock() throws IOException
+ {
+ if (!ParserType.STR_JSON.equals(parserType) || !useInputFormatApi) {
+ return;
+ }
+
+ final TimestampSpec timestampSpec = new TimestampSpec("t", null, null);
+ final DimensionsSpec dimensionsSpec = new DimensionsSpec(
+ ImmutableList.of(StringDimensionSchema.create("dim1PlusBar"))
+ );
+ final TransformSpec transformSpec = new TransformSpec(
+ null,
+ ImmutableList.of(new ExpressionTransform("dim1PlusBar", "concat(dim1 +
'bar')", TestExprMacroTable.INSTANCE))
+ );
+ final AggregatorFactory[] aggregatorFactories = {new
LongSumAggregatorFactory("met1", "met1")};
+ final GranularitySpec granularitySpec = new UniformGranularitySpec(
+ Granularities.DAY,
+ Granularities.HOUR,
+ true,
+ null
+ );
+ final DataSchema dataSchema = createDataSchema(
+ timestampSpec,
+ dimensionsSpec,
+ aggregatorFactories,
+ granularitySpec,
+ transformSpec
+ );
+
+ List<String> jsonBlockList = ImmutableList.of(
+ // include the line which can't be parsed into JSON object to form a
illegal json block
+ String.join("", STR_JSON_ROWS),
+
+ // exclude the last line to form a legal json block
+ String.join("", STR_JSON_ROWS.stream().limit(STR_JSON_ROWS.size() -
1).collect(Collectors.toList()))
Review comment:
could you re-trigger the CI ? The failure does not related to changes in
the PR.
```
Failed with bad exit code during 'Extraction'
[ERROR] Failed to execute goal on project druid-sql: Could not resolve
dependencies for project org.apache.druid:druid-sql:jar:0.21.0-SNAPSHOT: Failed
to collect dependencies at org.apache.calcite:calcite-core:jar:1.21.0 ->
org.codehaus.janino:janino:jar:3.0.11: Failed to read artifact descriptor for
org.codehaus.janino:janino:jar:3.0.11: Could not transfer artifact
org.codehaus.janino:janino-parent:pom:3.0.11 from/to central
(https://repo.maven.apache.org/maven2): Failed to transfer file
https://repo.maven.apache.org/maven2/org/codehaus/janino/janino-parent/3.0.11/janino-parent-3.0.11.pom
with status code 503 -> [Help 1]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
