hudi-agent commented on code in PR #19166:
URL: https://github.com/apache/hudi/pull/19166#discussion_r3521539211
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -208,4 +208,66 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase
{
}
}
}
+
+ test("Test column stats data skipping across multiple files with range, IN
and equality predicates") {
Review Comment:
🤖 nit: the test name starts with a capital "Test" while the other tests
added in this same PR (e.g. "test cow stream source…") use lowercase — could
you change it to "test column stats data skipping…" for consistency?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -508,6 +508,52 @@ class TestStructuredStreaming extends
HoodieSparkClientTestBase {
assertEquals(25, metaClient.getActiveTimeline.countInstants())
}
+ /**
+ * Injects a failing micro-batch into [[HoodieStreamingSink]] by pointing
the record key at a
+ * non-existent column so that every Hudi write throws. With
STREAMING_IGNORE_FAILED_BATCH
+ * enabled the sink must swallow the failure (unpersisting the write RDDs),
let the streaming
+ * query complete without crashing, and produce no commit.
+ */
+ @Test
+ def testStructuredStreamingIgnoreFailedBatch(): Unit = {
+ val (sourcePath, destPath) = initStreamingSourceAndDestPath("source",
"dest")
+ val records1 = recordsToStrings(dataGen.generateInsertsForPartition(
Review Comment:
🤖 nit: `records1` (and `inputDF1` two lines down) implies a
`records2`/`inputDF2` that never appears — could you drop the numeric suffix
and just use `records` / `inputDF`? The `1` in `coalesce(1)` on line 524 makes
it a bit confusing too.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
