wombatu-kun commented on code in PR #19389:
URL: https://github.com/apache/hudi/pull/19389#discussion_r3666023487
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderOnSpark.scala:
##########
@@ -243,13 +244,17 @@ class TestHoodieFileGroupReaderOnSpark extends
TestHoodieFileGroupReaderBase[Int
)
val deleteOpts: Map[String, String] = Map(
DELETE_KEY -> "op", DELETE_MARKER -> "d")
- val readOpts = if (mergeMode.equals("CUSTOM")) {
- fgReaderOpts ++ deleteOpts ++ Map(
- HoodieWriteConfig.WRITE_PAYLOAD_CLASS_NAME.key -> payloadClass)
+ val payloadOpts = if (mergeMode.equals("CUSTOM")) {
+ Map(HoodieWriteConfig.WRITE_PAYLOAD_CLASS_NAME.key -> payloadClass)
} else {
- fgReaderOpts ++ deleteOpts
+ Map.empty[String, String]
}
- val opts = readOpts
+ val opts = fgReaderOpts ++ deleteOpts ++ payloadOpts
+ // The write persists the marker on the table under the record-merge
property prefix. When the query does
+ // not restate the delete options, the table config is the only place the
reader can learn about them -
+ // which is what a query that just loads the path looks like.
+ val tableConfigOnly = markerFromTableConfigOnly.equals("true")
Review Comment:
Two layers hide it:
- The pre-existing params restate the delete options on the read, so
`DeleteContext` finds the plain delete field and marker keys directly in the
query properties and never has to un-prefix them out of the table config. The
path this PR fixes is not exercised at all.
- The params added here drop the delete options from the read, so the table
config becomes the reader's only source, but they still pass with and without
the fix: the Spark write path resolves the marker itself in
`HoodieCreateRecordUtils` and stamps `isDelete` at record creation, so those
rows reach the log already flagged and the reader is never asked to resolve the
marker.
So a Spark-level IT cannot fail on this in either shape.
`TestFileGroupReaderDeleteMarkerProps` in hudi-common drives the reader
directly, and 1 of its 4 cases fails without the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
