voonhous commented on code in PR #19458:
URL: https://github.com/apache/hudi/pull/19458#discussion_r3699399181
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -297,39 +290,9 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
*/
def hasSchemaOnRead: Boolean = internalSchemaOpt.isDefined
- /**
- * Data schema is determined as the actual schema of the Table's Data Files
(for ex, parquet/orc/etc);
- *
- * In cases when partition values are not persisted w/in the data files,
data-schema is defined as
- * <pre>table's schema - partition columns</pre>
- *
- * Check scala-doc for [[shouldExtractPartitionValuesFromPartitionPath]] for
more details
- */
- def dataSchema: StructType = if
(shouldExtractPartitionValuesFromPartitionPath) {
- prunePartitionColumns(tableStructSchema)
- } else {
- tableStructSchema
- }
-
- /**
- * Determines whether relation's schema could be pruned by Spark's Optimizer
- */
- def canPruneRelationSchema: Boolean =
- !HoodieTableMetadata.isMetadataTable(basePath.toString) &&
- (fileFormat.isInstanceOf[ParquetFileFormat] ||
fileFormat.isInstanceOf[OrcFileFormat]) &&
- // NOTE: In case this relation has already been pruned there's no point
in pruning it again
- prunedDataSchema.isEmpty &&
- // TODO(HUDI-5421) internal schema doesn't support nested schema pruning
currently
- !hasSchemaOnRead
-
override def sizeInBytes: Long = fileIndex.sizeInBytes
Review Comment:
Checked, this does not match the repo state:
- `BaseHoodieNestedSchemaPruning` and the
`Spark33NestedSchemaPruning`..`Spark42NestedSchemaPruning` family do not exist
on `master` (GitHub code search and `git grep` at `64afac03908b`: 0 hits).
There is no reflection dispatch in `customOptimizerRules`; `master` registers
the single rule with `new HoodieNestedSchemaPruning`, which is exactly the
registration this PR removes.
- The branch is one unrelated commit behind `master` (`64afac03908b`, a
Trino fix), not stale; the PR is mergeable.
- No `hudi-sparkX.Y.x` module references `canPruneRelationSchema` /
`dataSchema` / `updatePrunedDataSchema`: `git grep` finds 0 call sites outside
the deleted code, and the branch compiles under `-Dspark3.3`, `-Dspark3.5` and
`-Dspark4.2`.
- The `NestedSchemaPruning` code-search hits in the per-version parquet
readers are Spark's `SQLConf.NESTED_SCHEMA_PRUNING_ENABLED` conf key, not the
Hudi rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
