voonhous opened a new issue, #19465:
URL: https://github.com/apache/hudi/issues/19465
### Problem
After #19458, `latestCommitTimestamp` is the only field of
`HoodieTableState` (hudi-spark-common, `HoodieBaseRelation.scala`) that
anything still reads -- two
`.withLatestCommitTime(tableState.latestCommitTimestamp.orNull)` calls in
`HoodieMergeOnReadRDDV2.compute`.
The other seven fields (`tablePath`, `recordKeyField`, `orderingFields`,
`usesVirtualKeys`, `metadataConfig`, `recordMergeImplClasses`,
`recordMergeStrategyId`) are write-only: populated once in
`HoodieBaseRelation.tableState` and never read. The reads died in #17457
(FileGroupReader migration for all Spark datasource reads), #13544 and #12981;
`usesVirtualKeys` appears never to have been read at all. The whole object
(including a full `HoodieMetadataConfig` with its `Properties`) is a
non-transient field of `HoodieMergeOnReadRDDV2`, so it is serialized into every
MOR task for the sake of one `Option[String]`.
### Proposal
Delete `HoodieTableState` entirely and pass `latestCommitTimestamp` directly
to `HoodieMergeOnReadRDDV2`:
- `HoodieBaseRelation`: drop the `HoodieTableState` case class and the
`tableState` lazy val (the `recordMergerImpls` local and the
`fileIndex.getMetadataConfig` call there become dead too; the relation-level
`recordKeyField` / `orderingFields` lazy vals must stay -- they are still read
via `mandatoryFieldsForMerging` and the incremental relation traits)
- `HoodieMergeOnReadRDDV2`: replace the `tableState: HoodieTableState`
constructor parameter with `latestCommitTimestamp: Option[String]`
- Update the three `composeRDD` call sites (`MergeOnReadSnapshotRelation`,
`MergeOnReadIncrementalRelationV1/V2`)
Flagged by @wombatu-kun while reviewing #19458 ("worth dropping the other
seven by the same criterion, or noting that they are deliberately kept -
follow-up, not a blocker"); kept out of that PR to avoid growing an
already-reviewed diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
