voonhous commented on code in PR #19583:
URL: https://github.com/apache/hudi/pull/19583#discussion_r3766021618
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -148,12 +148,29 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
}
}
+ // The plain skip-merging reader cannot read a SHREDDED variant base file:
it requests native
+ // VariantType, which clips the shredded group to {metadata, value} and
reads value=null (the
+ // #19556 defect family). Such splits take the file-group reader below,
whose reader context
+ // requests the full-variant projection shape instead (#19578). Keyed off
the adapter building
+ // that shape rather than the mere presence of a variant column: it is None
below Spark 4.1,
+ // where the file-group reader would read the same nulls, so re-routing
there would cost the
+ // fast path for nothing.
+ private val shouldRerouteVariantSplit: Boolean =
+
sparkAdapter.buildFullVariantReadSchema(requiredSchema.structTypeSchema).isDefined
+
override def compute(split: Partition, context: TaskContext):
Iterator[InternalRow] = {
val partition = split.asInstanceOf[HoodieMergeOnReadPartition]
val bytesReadCallback =
HoodieSparkInputMetricsUtils.getFSBytesReadOnThreadCallback()
val iter: Iterator[InternalRow] = partition.split match {
- case dataFileOnlySplit if dataFileOnlySplit.logFiles.isEmpty =>
+ // A split whose partition values were parsed off the partition path
keeps the fast path even
+ // when re-routing would apply: only that reader appends them
(drop.partition.columns,
+ // extract-from-path, bootstrap fast read), and the file-group reader
branch below has no
+ // equivalent, so re-routing would trade null variants for null
partition columns. Those
+ // splits stay on the pre-existing behaviour; the same gap on the merged
branch is older than
+ // this change and is tracked separately.
Review Comment:
Filed it as #19594 and cited it, so the gap now has a number even though it
is fixed here rather than tracked. The PR description explains why it is
bundled: fix 1 re-routes base-only splits onto that same branch, so carving it
out would mean either shipping that regression or keeping a carve-out no test
could reach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
