hubgeter opened a new pull request, #70613:
URL: https://github.com/apache/doris/pull/70613
### Background
The Iceberg writer serializes partition values as strings for writer
grouping, partition paths, and FE commit metadata. The FE parses these strings
using each partition field's declared type before building Iceberg
PartitionData.
### What problem does this PR solve?
Issue Number: None
Related PR: #64315 (special-value handling), #65609 (round-trip
floating-point formatting)
PR Type: Bugfix
Problem Summary:
Finite FLOAT and DOUBLE identity partition values were serialized with
std::to_string, which retains only six digits after the decimal point. For
example, DOUBLE values 1.0000001 and 1.0000002 both became 1.000000, producing
incorrect partition metadata and potentially merging distinct partitions into
one writer. Subsequent partition filtering can silently omit matching rows.
Use the existing fmt shortest-round-trip formatting for finite partition
values while preserving the current NaN and infinity spellings. This fixes the
shared serialization used by writer keys, paths, and commit values. Previously
written incorrect metadata is not repaired by this change.
### Details
- **Frequency:** Triggered by dynamic FLOAT/DOUBLE identity partition values
that cannot round-trip through fixed six-decimal formatting; no concurrency is
required.
- **Root cause:** floating_point_partition_value_to_string used
std::to_string for finite values, losing precision before writer grouping and
FE partition parsing.
- **Why existing tests missed it:** Existing partition serialization tests
cover NaN and infinities, without finite precision or signed-zero bit
comparisons.
- **Fix:** Use fmt::format with the original FLOAT/DOUBLE type. Add actual
transform unit tests for adjacent values, small values, finite extrema,
subnormals, signed zeros, commit/human-string consistency, and distinct
partition strings.
- **Other paths and branches:** Dynamic writer keys, partition paths, and
commit values share this helper. No FE parser or Thrift changes are needed.
Full static partition fields use a separate precomputed-value path. Apache
branch-4.2 at 0a24ed86fb8da943e43facf653b5a3016470177a has the identical
affected production file and should receive this fix and tests; the complete
patch applies cleanly, and no additional prerequisite picks are needed.
- **Behavior change:** New writes preserve the original finite
floating-point partition values. NaN/infinity handling is unchanged. Existing
files with incorrect partition metadata still require separate recovery.
### Testing
- Passed clang-format v16, build-support/check-format.sh,
build-support/check-build-hygiene.sh, and git diff --check.
- Independently compiled helpers extracted from the baseline and modified
production source, then parsed their output using Java
Float.parseFloat/Double.parseDouble. The original implementation lost
finite-value precision in 10,006 FLOAT and 10,208 DOUBLE samples; the modified
implementation had zero round-trip failures across 40,034 samples, including
special values and signed zeros.
- The typed fast_float parser used by the new unit tests also preserved all
39,923 finite samples. These isolated experiments do not execute the full Doris
transform, writer, or commit path.
- Added PartitionTransformersTest.test_float_partition_value_round_trip and
test_double_partition_value_round_trip. Full BE UT was not run: this worktree
has no existing BE build, approximately 4.5 GB of free disk space remains, and
run-be-ut.sh builds the complete BE test binary even with a GTest filter.
- No end-to-end Iceberg SQL regression or clang-tidy run: an isolated built
cluster and BE compilation database are not available. No regression golden
files were handwritten.
- Verified that the production fix and unit-test patch apply cleanly to the
current branch-4.2 files.
### Notes for Reviewers
This remains a draft until the full BE tests and external Iceberg SQL checks
are completed. Suggested unit-test command: ./run-be-ut.sh --run
--filter='PartitionTransformersTest.*' -j 1. The SQL check should write DOUBLE
partitions 1.0000001 and 1.0000002, verify distinct accurate typed partitions
in the files metadata, and expect equality/range counts of 1, 1, and 2.
Recommend picking this PR to branch-4.2 after merge.
### Release note
Fix precision loss when writing finite FLOAT/DOUBLE Iceberg identity
partition values, preventing incorrect partition metadata and missing rows
during partition filtering for newly written data.
### Check List (For Author)
- Test:
- [ ] Regression test
- [ ] Unit Test (added; full BE test binary not built locally)
- [x] Manual test (isolated serialization/parser experiments and
formatting/hygiene checks described above)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: full BE UT, end-to-end Iceberg
SQL regression, and clang-tidy remain pending; keep as Draft.
### AI Disclosure
- Tool: Codex
- Generated: Code, unit tests, validation probes, and this description.
- Human review: Pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]