FrankChen021 commented on a change in pull request #10383:
URL: https://github.com/apache/druid/pull/10383#discussion_r520275473
##########
File path:
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaIndexTaskTest.java
##########
@@ -2726,4 +2726,83 @@ public void close()
null
);
}
+
+ @Test(timeout = 60_000L)
+ public void testMultipleLinesJSONText() throws Exception
+ {
+ reportParseExceptions = false;
+ maxParseExceptions = 1000;
+ maxSavedParseExceptions = 2;
+
+ // Insert data
+ try (final KafkaProducer<byte[], byte[]> kafkaProducer =
kafkaServer.newProducer()) {
+ kafkaProducer.initTransactions();
+ kafkaProducer.beginTransaction();
+
+ //multiple objects in one Kafka record will yield 2 rows in druid
+ String wellformed = toJsonString(true, "2049", "d2", "y", "10", "22.0",
"2.0") +
+ toJsonString(true, "2049", "d3", "y", "10", "23.0",
"3.0");
+
+ //multiple objects in one Kafka record but some objects are in
ill-formed format
+ //the whole ProducerRecord will be discarded
+ String illformed = "{\"timestamp\":2049, \"dim1\": \"d4\",
\"dim2\":\"x\", \"dimLong\": 10, \"dimFloat\":\"24.0\", \"met1\":\"2.0\" }" +
+ "{\"timestamp\":2049, \"dim1\": \"d5\", \"dim2\":\"y\",
\"dimLong\": 10, \"dimFloat\":\"24.0\", \"met1\":invalidFormat }" +
+ "{\"timestamp\":2049, \"dim1\": \"d6\", \"dim2\":\"z\",
\"dimLong\": 10, \"dimFloat\":\"24.0\", \"met1\":\"3.0\" }";
+
+ ProducerRecord[] producerRecords = new ProducerRecord[]{
+ // pretty formatted
+ new ProducerRecord<>(topic, 0, null, jb(true, "2049", "d1", "y",
"10", "20.0", "1.0")),
+ //well-formed
+ new ProducerRecord<>(topic, 0, null, StringUtils.toUtf8(wellformed)),
+ //ill-formed
+ new ProducerRecord<>(topic, 0, null,
StringUtils.toUtf8("illformed")),
Review comment:
Yes, `illformed` here should be a variable instead of a text string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
