JustinBinber opened a new pull request, #11076:
URL: https://github.com/apache/paimon/pull/11076

   ### Purpose
   
   Fixes #11075.
   
   Avoid shuffling one record per null input when building scalar BTree/Bitmap 
global indexes, without moving the overhead onto ordinary non-null inputs.
   
   - Select compression automatically at an observed null ratio of at least 
80%. Spark reuses original-row range sampling and deduplicates source-partition 
counts when the sampler rescans skewed inputs. Flink observes bounded windows 
and adapts as density changes.
   - Keep Spark sampling and the uncompressed exchange on native internal rows 
with SQL's compact unsafe-row serializer. Flink's optional-payload encoding 
sends ordinary rows in the original three-field representation, without a 
fourth-field penalty.
   - Buffer at most 4,096 null row IDs, retain assigned destinations, and 
expand before local sorting. Composite/Multivalue paths, persisted index 
formats and query code remain unchanged.
   - Validate serializer snapshots and reject malformed record lengths or 
unsupported snapshot versions.
   
   ### Tests
   
   Validated with Java 8, Scala 2.12, Spark 3 and Flink 1. Formatting, 
Checkstyle and enforcer checks were enabled on final test runs.
   
   ```sh
   mvn -pl paimon-codegen-loader -am -DskipTests package
   mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
     -DwildcardSuites=none -DfailIfNoTests=false \
     
-Dtest=NullBitmapBufferTest,NullBitmapShuffleOperatorTest,NullBitmapTransportTypeInfoTest,SortedIndexTopoBuilderTest,SortedGlobalIndexITCase,RangeShuffleTest
 test
   mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
     -DfailIfNoTests=false \
     
-Dtest=NullBitmapBufferTest,NullBitmapShuffleTest,NullBitmapAdaptiveTest,NullBitmapShuffleBenchmarkTest,NullBitmapIndexComparisonTest,SortedIndexTopoBuilderTest
 \
     
-DwildcardSuites=org.apache.paimon.spark.procedure.NullBitmapGlobalIndexTest,org.apache.paimon.spark.procedure.CreateGlobalIndexProcedureTest
 test
   ```
   
   The codegen packaging step is needed on a fresh checkout because its 
isolated runtime classes are unpacked during `prepare-package`.
   
   Coverage includes automatic threshold decisions, oversized source-partition 
resampling, empty/single-output inputs, mixed build tasks, string/decimal/date 
keys, both bypass and serialized shuffle writers, adaptive density changes, 
transport snapshot round trips, incremental append/refresh, and 
equality/range/null predicates.
   
   Twelve shared-boundary layouts for each of BTree and Bitmap produce 
identical ordered rows, file counts, metadata, file sizes and file contents. 
All 96 paired predicate checks have identical cold read bytes. Shared 
boundaries isolate compression from the natural variance of independent range 
sampling.
   
   A separate warmed benchmark alternates execution order and reports three-run 
medians for elapsed time, executor CPU, GC, spill and actual shuffle-write 
metrics. Low-null/variable-width cases use two million rows; other cases use 
200,000 rows. Results are local, not a claim of production latency improvement.
   
   Spark actual shuffle metrics, original/adaptive (three alternating warmed 
runs; local execution):
   
   | Nulls / key | Rows | Shuffle bytes | Byte change | Elapsed ms | Executor 
CPU ms |
   |---|---:|---:|---:|---:|---:|
   | 0% / INT | 2,000,000 | 16,808,014 / 16,807,351 | -0.004% | 882 / 850 | 
1125 / 1171 |
   | 2% / INT | 2,000,000 | 16,757,408 / 16,764,942 | +0.045% | 581 / 599 | 890 
/ 828 |
   | 10% / INT | 200,000 | 1,654,782 / 1,654,709 | -0.004% | 165 / 135 | 125 / 
93 |
   | 50% / INT | 200,000 | 1,363,318 / 1,365,177 | +0.136% | 118 / 137 | 125 / 
109 |
   | 79% / INT | 200,000 | 1,173,760 / 1,175,438 | +0.143% | 115 / 114 | 125 / 
125 |
   | 80% / INT | 200,000 | 1,165,262 / 608,809 | -47.753% | 145 / 227 | 125 / 
234 |
   | 90% / INT | 200,000 | 1,103,612 / 305,665 | -72.303% | 93 / 141 | 109 / 
125 |
   | 100% / INT | 200,000 | 1,010,977 / 2,812 | -99.722% | 96 / 83 | 109 / 93 |
   | 0% / STRING | 2,000,000 | 19,865,834 / 19,872,059 | +0.031% | 955 / 922 | 
1812 / 1718 |
   
   Spill was zero in these runs. Local timing varies, especially for the 
200k-row cases; dense inputs trade additional aggregation CPU for substantially 
less network traffic. These numbers are not a claim of universal end-to-end 
speedup.
   
   Flink's actual tuple-serializer comparison over 20,000 rows uses exactly the 
original byte count at 0%, 2% and 50% nulls (1,520,000 bytes before/after). At 
90% it drops to 171,292 bytes, and all-null input to 20,036 bytes, including 
the initial uncompressed observation window. These are serialized payload 
counts, not cluster network metrics.
   
   
   Full-repository RAT currently reports one pre-existing, unchanged 
compatibility fixture without a recognized license: 
`paimon-flink/paimon-flink-common/src/test/resources/compatibility/pending-splits-incremental-v1-chain-v2`.
 New files have Apache headers; that unrelated fixture is not changed by this 
PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to