sumitsingh-in commented on code in PR #20183:
URL: https://github.com/apache/hudi/pull/20183#discussion_r4181503519
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaEvolutionUtils.scala:
##########
@@ -84,12 +87,8 @@ class ParquetSchemaEvolutionUtils(sharedConf: Configuration,
// Clone new conf
val hadoopAttemptConf = new Configuration(sharedConf)
typeChangeInfos = if (shouldUseInternalSchema) {
- // Empty projections (count(*), select 1) read no column data, so there
is nothing to
- // reconstruct - and querySchemaOption is the UNPRUNED table schema in
that case (see
- // pruneInternalSchema), so running the guard would fail queries that
work fine.
- if (requiredSchema.nonEmpty) {
Review Comment:
- For an empty projection (`count(*)`, `select 1`, partition-only
selects), `pruneInternalSchema` returns the unpruned table schema.
`ParquetSchemaEvolutionUtils.getHadoopConfClone` merged it and set it as
the requested read schema, so a count asked the base-file reader for every
table column.
- The internal-schema converter has no VARIANT arm, so the variant column
was requested as `struct<metadata, value>`. Spark 4.x's vectorized reader
requires a VARIANT-annotated Parquet group to be read as
`VariantType`, and fails while initializing:
```
org.apache.spark.sql.AnalysisException: Invalid Spark read type:
expected optional group v (VARIANT(1)) {
required binary metadata;
optional binary value;
} to be variant type but found STRUCT<metadata: BINARY NOT NULL, value:
BINARY NOT NULL>
at
...ParquetToSparkSchemaConverter.convertGroupField(ParquetSchemaConverter.scala:407)
at
...VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:199)
at
...Spark42ParquetReader.buildVectorizedIterator(Spark42ParquetReader.scala:235)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
