lokeshj1703 opened a new pull request, #19637: URL: https://github.com/apache/hudi/pull/19637
### Describe the issue this Pull Request addresses Closes #19636. Supersedes #19773 (consolidated here). Null values in configured ordering columns were converted to the integer default sentinel in the common and Spark read paths. That conflated missing field values with intentional natural-order records and could compare an Integer sentinel against a Long ordering value. This revision preserves null values in configured ordering fields. A null base (existing) ordering value ranks lowest, so a record carrying a real ordering value wins the merge, while a null incoming ordering value or an incompatible type still fails the comparison. Following the subsequent refinement, an empty or absent ordering-field collection continues returning Integer(0), retaining existing natural-order behavior. DELETE sentinels also remain unchanged. ### Summary and Changelog - Preserve null field values in both RecordContext ordering-field overloads while retaining the default Integer(0) when no ordering fields are configured. - Preserve nulls in Spark/Flink engine and Java ordering-value accessors. Keep the existing default for absent ordering fields across Spark, Flink, and Avro indexed records. - Make common and Spark ordering converters null-safe, including null components of multi-field ordering values. Centralize Flink null handling in RowDataUtils.flinkValFunc so OrderingValueEngineTypeConverter delegates directly without duplicate null guards. - Guard the event-time record mergers so a null base ordering value ranks lowest and the incoming record wins, while a null incoming value or an incompatible type still fails. The partial-update final merge gets the same null-base guard. No mixed-class natural-order fallback is added. - Preserve HoodieEmptyRecord and legacy Integer(0) delete sentinels. Make BufferedRecord.isCommitTimeOrderingDelete delegate to OrderingValues.isCommitTimeOrderingValue so null ordering values in native-log delete records are also treated as commit-time deletes. Document that this predicate is only for delete records; a null incoming ordering value in ordinary data records remains subject to event-time comparison failures. Keep maxOrderingValue as a direct comparison. - Add regression coverage for AVRO/SPARK/FLINK extraction, DELETE sentinels, null buffer serialization, converters, and commit-time versus event-time merge behavior. Verify that empty ordering-field lists and arrays yield Integer(0) and keep the newer record through both final and delta merges. Cover the base-null merge behavior directly at the merger level, including the partial-update final merge where a null base value keeps the incoming record, and extend the end-to-end regression to expect a null base ordering value to succeed with the incoming record winning across COW and MOR while a null incoming value fails. Validation on Java 17: 89 focused tests passed (29 common, 7 Spark client, 21 Spark datasource, and 32 Flink client), using Spark 3.5.5 and Flink 2.2.1. This includes all 12 end-to-end null-ordering cases and the existing Spark null-precombine tests. Checkstyle, Apache RAT, and the repository PR-body compliance check passed. The full ScalaTest and cross-version suites were not completed. After centralizing the Flink conversion guard, 71 focused Flink tests passed, including 36 direct null-conversion cases covering supported scalar types and both timezone settings. Existing non-null conversions and single-/multi-field ordering tests also passed. After the final common-code simplifications, all 29 focused common tests passed again, including maxOrderingValue rejecting nulls and accepting the default ordering value. Redundant null guards in RecordContext delegate to the existing null-safe conversion methods. After reintroducing the base-null merge tolerance, the merger and end-to-end tests passed on Java 11 with Spark 3.5: TestBufferedRecordMergerFactory (9 cases), TestBufferedRecordMerger (19 cases, including the partial-update null-base success), and the end-to-end TestNullOrderingValueMerge (24 cases across COW and MOR, AVRO and SPARK). After correcting null-delete detection, 144 targeted tests passed on Java 17: 64 shared/native-log regressions, all 26 Flink reader tests, and 54 Spark tests covering reader behavior, default/custom mergers, clustering, time travel, and record-index deletes. Spark and Flink reader assertions now distinguish empty ordering-field lists (Integer(0)) from missing or null configured values (null). The full Trino and Spark 4.2 suites were not rerun locally; reported runtime failures were traced to the shared delete predicate. The latest Spark/Flink runs passed style checks and skipped RAT because of existing generated local metastore files. ### Impact Records with configured ordering fields retain null field values instead of acquiring an integer sentinel. Records without ordering fields continue using Integer(0): equal default values keep the newer record in the standard merger. This does not change the configured merge mode or add a fallback for mismatched types. Event-time merges tolerate a null base ordering value, with the incoming record winning; a null incoming value or a mismatched type still fails. DELETE-statement behavior and legacy sentinel recognition are preserved. No public API signature or storage-format changes. ### Risk Level medium The extraction changes affect shared reader code and multiple engines. Focused tests cover null and legacy sentinel handling, existing merger behavior, and end-to-end writes. Full project and cross-version engine suites have not been run. ### Documentation Update none - This corrects internal null handling without adding configuration or relaxing event-time ordering requirements. Method documentation distinguishes absent ordering fields from null field values. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
