hudi-agent commented on code in PR #19458:
URL: https://github.com/apache/hudi/pull/19458#discussion_r3699194546
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -297,39 +290,9 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
*/
def hasSchemaOnRead: Boolean = internalSchemaOpt.isDefined
- /**
- * Data schema is determined as the actual schema of the Table's Data Files
(for ex, parquet/orc/etc);
- *
- * In cases when partition values are not persisted w/in the data files,
data-schema is defined as
- * <pre>table's schema - partition columns</pre>
- *
- * Check scala-doc for [[shouldExtractPartitionValuesFromPartitionPath]] for
more details
- */
- def dataSchema: StructType = if
(shouldExtractPartitionValuesFromPartitionPath) {
- prunePartitionColumns(tableStructSchema)
- } else {
- tableStructSchema
- }
-
- /**
- * Determines whether relation's schema could be pruned by Spark's Optimizer
- */
- def canPruneRelationSchema: Boolean =
- !HoodieTableMetadata.isMetadataTable(basePath.toString) &&
- (fileFormat.isInstanceOf[ParquetFileFormat] ||
fileFormat.isInstanceOf[OrcFileFormat]) &&
- // NOTE: In case this relation has already been pruned there's no point
in pruning it again
- prunedDataSchema.isEmpty &&
- // TODO(HUDI-5421) internal schema doesn't support nested schema pruning
currently
- !hasSchemaOnRead
-
override def sizeInBytes: Long = fileIndex.sizeInBytes
Review Comment:
🤖 This branch looks stale against `main`: the single
`HoodieNestedSchemaPruning` (and the `new HoodieNestedSchemaPruning`
registration this diff removes) no longer exists there. `main` now dispatches
via reflection in `customOptimizerRules` to a version-specific family
(`Spark33`…`Spark42NestedSchemaPruning` extending
`BaseHoodieNestedSchemaPruning`), and those rules still call
`relation.canPruneRelationSchema`, `relation.dataSchema`, and
`relation.updatePrunedDataSchema`. Removing these three members here will break
compilation of every `hudi-sparkX.Y.x` module once rebased. If the dead-code
premise holds, those rules and their registration need to be deleted too; could
you rebase and extend the removal accordingly? @yihua could you confirm the
intended target state here?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/BaseHoodieCatalystPlanUtils.scala:
##########
@@ -22,24 +22,17 @@ import org.apache.hudi.SparkAdapterSupport
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.{ResolvedTable,
TableOutputResolver}
import org.apache.spark.sql.catalyst.catalog.CatalogStorageFormat
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet,
Expression, ProjectionOverSchema}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
import org.apache.spark.sql.catalyst.plans.JoinType
import org.apache.spark.sql.catalyst.plans.logical.{CreateIndex, DropIndex,
HoodieShowIndexes, InsertIntoStatement, Join, JoinHint, LogicalPlan,
RefreshIndex}
import org.apache.spark.sql.connector.catalog.{Identifier, Table, TableCatalog}
import org.apache.spark.sql.execution.{ExtendedMode, SimpleMode}
import org.apache.spark.sql.execution.command.{CreateTableLikeCommand,
ExplainCommand, RepairTableCommand}
import org.apache.spark.sql.execution.datasources.LogicalRelation
Review Comment:
🤖 Removing `projectOverSchema` from this trait will also break the
per-version overrides that still exist on `main`:
`HoodieSpark35CatalystPlanUtils` / `HoodieSpark42CatalystPlanUtils` (and the
other versions) declare `override def projectOverSchema(...)`, and
`BaseHoodieNestedSchemaPruning.prunePhysicalColumns` calls
`planUtils.projectOverSchema(...)`. Those overrides won't compile against a
base trait that no longer declares the method. Same root cause as the
`HoodieBaseRelation` comment — this needs to be part of the same rebased
cleanup.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
