JustinBinber opened a new pull request, #11076:
URL: https://github.com/apache/paimon/pull/11076
### Purpose
Fixes #11075.
Avoid shuffling one record per null input when building scalar BTree/Bitmap
global indexes, without moving the overhead onto ordinary non-null inputs.
- Select compression automatically at an observed null ratio of at least
80%. Spark reuses original-row range sampling and deduplicates source-partition
counts when the sampler rescans skewed inputs. Flink observes bounded windows
and adapts as density changes.
- Keep Spark sampling and the uncompressed exchange on native internal rows
with SQL's compact unsafe-row serializer. Flink's optional-payload encoding
sends ordinary rows in the original three-field representation, without a
fourth-field penalty.
- Buffer at most 4,096 null row IDs, retain assigned destinations, and
expand before local sorting. Composite/Multivalue paths, persisted index
formats and query code remain unchanged.
- Validate serializer snapshots and reject malformed record lengths or
unsupported snapshot versions.
### Tests
Validated with Java 8, Scala 2.12, Spark 3 and Flink 1. Formatting,
Checkstyle and enforcer checks were enabled on final test runs.
```sh
mvn -pl paimon-codegen-loader -am -DskipTests package
mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
-DwildcardSuites=none -DfailIfNoTests=false \
-Dtest=NullBitmapBufferTest,NullBitmapShuffleOperatorTest,NullBitmapTransportTypeInfoTest,SortedIndexTopoBuilderTest,SortedGlobalIndexITCase,RangeShuffleTest
test
mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
-DfailIfNoTests=false \
-Dtest=NullBitmapBufferTest,NullBitmapShuffleTest,NullBitmapAdaptiveTest,NullBitmapShuffleBenchmarkTest,NullBitmapIndexComparisonTest,SortedIndexTopoBuilderTest
\
-DwildcardSuites=org.apache.paimon.spark.procedure.NullBitmapGlobalIndexTest,org.apache.paimon.spark.procedure.CreateGlobalIndexProcedureTest
test
```
The codegen packaging step is needed on a fresh checkout because its
isolated runtime classes are unpacked during `prepare-package`.
Coverage includes automatic threshold decisions, oversized source-partition
resampling, empty/single-output inputs, mixed build tasks, string/decimal/date
keys, both bypass and serialized shuffle writers, adaptive density changes,
transport snapshot round trips, incremental append/refresh, and
equality/range/null predicates.
Twelve shared-boundary layouts for each of BTree and Bitmap produce
identical ordered rows, file counts, metadata, file sizes and file contents.
All 96 paired predicate checks have identical cold read bytes. Shared
boundaries isolate compression from the natural variance of independent range
sampling.
A separate warmed benchmark alternates execution order and reports three-run
medians for elapsed time, executor CPU, GC, spill and actual shuffle-write
metrics. Low-null/variable-width cases use two million rows; other cases use
200,000 rows. Results are local, not a claim of production latency improvement.
Spark actual shuffle metrics, original/adaptive (three alternating warmed
runs; local execution):
| Nulls / key | Rows | Shuffle bytes | Byte change | Elapsed ms | Executor
CPU ms |
|---|---:|---:|---:|---:|---:|
| 0% / INT | 2,000,000 | 16,808,014 / 16,807,351 | -0.004% | 882 / 850 |
1125 / 1171 |
| 2% / INT | 2,000,000 | 16,757,408 / 16,764,942 | +0.045% | 581 / 599 | 890
/ 828 |
| 10% / INT | 200,000 | 1,654,782 / 1,654,709 | -0.004% | 165 / 135 | 125 /
93 |
| 50% / INT | 200,000 | 1,363,318 / 1,365,177 | +0.136% | 118 / 137 | 125 /
109 |
| 79% / INT | 200,000 | 1,173,760 / 1,175,438 | +0.143% | 115 / 114 | 125 /
125 |
| 80% / INT | 200,000 | 1,165,262 / 608,809 | -47.753% | 145 / 227 | 125 /
234 |
| 90% / INT | 200,000 | 1,103,612 / 305,665 | -72.303% | 93 / 141 | 109 /
125 |
| 100% / INT | 200,000 | 1,010,977 / 2,812 | -99.722% | 96 / 83 | 109 / 93 |
| 0% / STRING | 2,000,000 | 19,865,834 / 19,872,059 | +0.031% | 955 / 922 |
1812 / 1718 |
Spill was zero in these runs. Local timing varies, especially for the
200k-row cases; dense inputs trade additional aggregation CPU for substantially
less network traffic. These numbers are not a claim of universal end-to-end
speedup.
Flink's actual tuple-serializer comparison over 20,000 rows uses exactly the
original byte count at 0%, 2% and 50% nulls (1,520,000 bytes before/after). At
90% it drops to 171,292 bytes, and all-null input to 20,036 bytes, including
the initial uncompressed observation window. These are serialized payload
counts, not cluster network metrics.
Full-repository RAT currently reports one pre-existing, unchanged
compatibility fixture without a recognized license:
`paimon-flink/paimon-flink-common/src/test/resources/compatibility/pending-splits-incremental-v1-chain-v2`.
New files have Apache headers; that unrelated fixture is not changed by this
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]