hubgeter opened a new pull request, #70613:
URL: https://github.com/apache/doris/pull/70613

   ### Background
   
   The Iceberg writer serializes partition values as strings for writer 
grouping, partition paths, and FE commit metadata. The FE parses these strings 
using each partition field's declared type before building Iceberg 
PartitionData.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #64315 (special-value handling), #65609 (round-trip 
floating-point formatting)
   PR Type: Bugfix
   
   Problem Summary:
   
   Finite FLOAT and DOUBLE identity partition values were serialized with 
std::to_string, which retains only six digits after the decimal point. For 
example, DOUBLE values 1.0000001 and 1.0000002 both became 1.000000, producing 
incorrect partition metadata and potentially merging distinct partitions into 
one writer. Subsequent partition filtering can silently omit matching rows.
   
   Use the existing fmt shortest-round-trip formatting for finite partition 
values while preserving the current NaN and infinity spellings. This fixes the 
shared serialization used by writer keys, paths, and commit values. Previously 
written incorrect metadata is not repaired by this change.
   
   ### Details
   
   - **Frequency:** Triggered by dynamic FLOAT/DOUBLE identity partition values 
that cannot round-trip through fixed six-decimal formatting; no concurrency is 
required.
   - **Root cause:** floating_point_partition_value_to_string used 
std::to_string for finite values, losing precision before writer grouping and 
FE partition parsing.
   - **Why existing tests missed it:** Existing partition serialization tests 
cover NaN and infinities, without finite precision or signed-zero bit 
comparisons.
   - **Fix:** Use fmt::format with the original FLOAT/DOUBLE type. Add actual 
transform unit tests for adjacent values, small values, finite extrema, 
subnormals, signed zeros, commit/human-string consistency, and distinct 
partition strings.
   - **Other paths and branches:** Dynamic writer keys, partition paths, and 
commit values share this helper. No FE parser or Thrift changes are needed. 
Full static partition fields use a separate precomputed-value path. Apache 
branch-4.2 at 0a24ed86fb8da943e43facf653b5a3016470177a has the identical 
affected production file and should receive this fix and tests; the complete 
patch applies cleanly, and no additional prerequisite picks are needed.
   - **Behavior change:** New writes preserve the original finite 
floating-point partition values. NaN/infinity handling is unchanged. Existing 
files with incorrect partition metadata still require separate recovery.
   
   ### Testing
   
   - Passed clang-format v16, build-support/check-format.sh, 
build-support/check-build-hygiene.sh, and git diff --check.
   - Independently compiled helpers extracted from the baseline and modified 
production source, then parsed their output using Java 
Float.parseFloat/Double.parseDouble. The original implementation lost 
finite-value precision in 10,006 FLOAT and 10,208 DOUBLE samples; the modified 
implementation had zero round-trip failures across 40,034 samples, including 
special values and signed zeros.
   - The typed fast_float parser used by the new unit tests also preserved all 
39,923 finite samples. These isolated experiments do not execute the full Doris 
transform, writer, or commit path.
   - Added PartitionTransformersTest.test_float_partition_value_round_trip and 
test_double_partition_value_round_trip. Full BE UT was not run: this worktree 
has no existing BE build, approximately 4.5 GB of free disk space remains, and 
run-be-ut.sh builds the complete BE test binary even with a GTest filter.
   - No end-to-end Iceberg SQL regression or clang-tidy run: an isolated built 
cluster and BE compilation database are not available. No regression golden 
files were handwritten.
   - Verified that the production fix and unit-test patch apply cleanly to the 
current branch-4.2 files.
   
   ### Notes for Reviewers
   
   This remains a draft until the full BE tests and external Iceberg SQL checks 
are completed. Suggested unit-test command: ./run-be-ut.sh --run 
--filter='PartitionTransformersTest.*' -j 1. The SQL check should write DOUBLE 
partitions 1.0000001 and 1.0000002, verify distinct accurate typed partitions 
in the files metadata, and expect equality/range counts of 1, 1, and 2. 
Recommend picking this PR to branch-4.2 after merge.
   
   ### Release note
   
   Fix precision loss when writing finite FLOAT/DOUBLE Iceberg identity 
partition values, preventing incorrect partition metadata and missing rows 
during partition filtering for newly written data.
   
   ### Check List (For Author)
   
   - Test:
       - [ ] Regression test
       - [ ] Unit Test (added; full BE test binary not built locally)
       - [x] Manual test (isolated serialization/parser experiments and 
formatting/hygiene checks described above)
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: full BE UT, end-to-end Iceberg 
SQL regression, and clang-tidy remain pending; keep as Draft.
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: Code, unit tests, validation probes, and this description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to