yihua opened a new pull request, #19388:
URL: https://github.com/apache/hudi/pull/19388
### Describe the issue this Pull Request addresses
When schema conflict resolution is enabled and a writer commits a batch
whose writer schema is the Avro null schema (for example, an ingestion round
that writes no data),
`SimpleSchemaConflictResolutionStrategy.resolveConcurrentSchemaEvolution`
resolves the table schema at the current transaction's owner instant. At
pre-commit time that instant is inflight and has no completion time, so the
completion-time filter in
`ConcurrentSchemaEvolutionTableSchemaGetter.getLastCommitMetadataWithValidSchemaFromTimeline`
calls `String.compareTo(null)`, and the commit fails:
```
Caused by: org.apache.hudi.exception.HoodieException: Unable to get table
schema
at
org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy.getTableSchemaAtInstant(SimpleSchemaConflictResolutionStrategy.java:173)
at
org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy.resolveConcurrentSchemaEvolution(SimpleSchemaConflictResolutionStrategy.java:74)
...
Caused by: java.lang.NullPointerException
at java.lang.String.compareTo(String.java:1155)
at
org.apache.hudi.common.table.timeline.InstantComparison.lambda$static$3(InstantComparison.java:34)
at
org.apache.hudi.common.table.timeline.InstantComparison.compareTimestamps(InstantComparison.java:38)
at
org.apache.hudi.client.transaction.ConcurrentSchemaEvolutionTableSchemaGetter.lambda$getLastCommitMetadataWithValidSchemaFromTimeline$3(ConcurrentSchemaEvolutionTableSchemaGetter.java:175)
```
The defect is twofold, and Hudi 1.x writes both table versions:
- On table version 6, ordering the schema evolution timeline by completion
time is wrong to begin with: version 6 does not record completion time on disk
(the in-memory value is synthesized from the meta file modification time), and
0.x writers concurrently writing the same table order the timeline by requested
time. Requested-time ordering restores 0.x parity and also removes the NPE on
this version, since every instant carries a requested time.
- On table version 8 and above, completion-time ordering is correct, but the
same NPE fires for an empty-batch commit: the current transaction's owner
instant is not completed yet and has no completion time.
A transaction with a null writer schema does not evolve the schema, so the
resolution should adopt the current table schema.
### Summary and Changelog
- The per-timeline-version instant ordering is exposed as a first-class API
on `InstantComparator`: `orderingComparator()` / `getOrderingTime(instant)`
(requested-time based in v1, completion-time based in v2), so version handling
lives in the timeline layer rather than in callers.
- `ConcurrentSchemaEvolutionTableSchemaGetter`: sorts and bounds the schema
evolution timeline with that ordering (completion time for table version 8 and
above, requested time for earlier versions, matching 0.x). A target instant
without an ordering time (not completed yet, on table version 8 and above) does
not bound the lookup instead of throwing NPE.
- `SimpleSchemaConflictResolutionStrategy`: the null-writer-schema path
adopts the table schema as of the current transaction's owner instant. On table
version 6 the lookup is bounded by the instant's requested time (matching 0.x
behavior); on table version 8 and above the inflight instant carries no
completion time, so the latest table schema is resolved. The prior-instant
lookup for the transaction start snapshot also uses the version-appropriate
ordering time.
- Tests: `TestSimpleSchemaConflictResolutionStrategy` gains the
inflight-instant NPE regression plus table-version-6 cases proving
requested-time bounding; `TestConcurrentSchemaEvolutionTableSchemaGetter` gains
latest-schema ordering regressions with the same two-commit layout on both
versions (the earlier-requested commit completes last), so table version 8 and
above must return the completion-time winner and table version 6 must return
the requested-time winner even when the synthesized completion order disagrees;
`TestInstantComparators` covers the new `InstantComparator` ordering APIs
directly.
### Impact
Fixes commit failures on multi-writer tables for ingestion rounds that write
no data, on table version 6 and table version 8 and above alike. No public API
or user-facing behavior change otherwise.
### Risk Level
low
The new regression tests fail with the production NPE signature before the
fix and pass after; the table-version-6 ordering tests fail without the
version-aware ordering and pass with it; both full test classes pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
