voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3901763668
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -149,13 +149,15 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
}
}
- // The plain skip-merging reader cannot read a SHREDDED variant base file:
it requests native
- // VariantType, which clips the shredded group to {metadata, value} and
reads value=null (the
- // #19556 defect family). Such splits take the file-group reader below,
whose reader context
- // requests the full-variant projection shape instead (#19578). Keyed off
the adapter building
- // that shape rather than the mere presence of a variant column: it is None
below Spark 4.1,
- // where the file-group reader would read the same nulls, so re-routing
there would cost the
- // fast path for nothing.
+ // A split whose required schema has a top-level variant column takes the
file-group reader
+ // below, whose reader context requests the full-variant projection shape
for parquet base
+ // files (#19578), so a SHREDDED base file is read on this legacy path
through the same
+ // contract as everywhere else. Without a top-level variant the base-only
split stays on
+ // requiredSchemaReaderSkipMerging, whose native VariantType request the
Spark 4.1+ row reader
Review Comment:
Right: the legacy file format takes `enableVectorizedReader` straight from
the inherited `ParquetFileFormat.supportBatch`, and `VariantType` is atomic, so
that split reads through the vectorized reader unless the user turns it off.
The comment here and the scaladoc in `TestStreamingSource` now say "parquet
reader (the vectorized one at stock settings)" and name where the veto actually
lives.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -22,13 +22,11 @@ package org.apache.spark.sql.hudi.dml.schema
import org.apache.hudi.{DataSourceReadOptions, HoodieSparkUtils}
import org.apache.hudi.common.avro.VariantShreddingRuntime
import org.apache.hudi.common.fs.FSUtils
-import org.apache.hudi.common.model.HoodieLogFile
import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType
import org.apache.hudi.common.model.WriteOperationType
import org.apache.hudi.common.schema.HoodieSchema
import org.apache.hudi.common.schema.internal.HoodieSchemaException
import org.apache.hudi.common.table.TableSchemaResolver
-import org.apache.hudi.common.table.log.HoodieLogFormat
import org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType
import org.apache.hudi.common.testutils.HoodieTestUtils
import org.apache.hudi.common.util.StringUtils
Review Comment:
Dropped both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
