JingsongLi commented on PR #11076:
URL: https://github.com/apache/paimon/pull/11076#issuecomment-6785720722

   The explicit serializer fixes the Kryo registration issue; I rechecked the 
strict-registration case successfully on `8e646d5c4b`.
   
   Before enabling this optimization by default, I suggest two follow-ups:
   
   1. **Add a shared Spark/Flink build option that restores the original 
execution path when disabled.** For example, 
`sorted-index.null-bitmap-shuffle.enabled` (proposed name): `true` permits the 
existing adaptive compression, while `false` restores Spark's original SQL 
`repartitionByRange` path and Flink's original three-field schema without the 
compression/expansion operators or custom transport serializer. Merely passing 
`compress=false` to `NullBitmapShuffle.sort` is insufficient because that still 
uses the new `sortUncompressed` implementation. The switch should provide a way 
to isolate and avoid regressions in the new shuffle implementation, as well as 
compression overhead. Given the current performance evidence, I would prefer an 
opt-in default initially.
   
   2. **Remove the bitmap payload field before Flink's local sort.** 
`NullBitmapExpandOperator` currently emits a four-field value even after 
expansion, and `SortUtils` combines it with the three-field sort key. The 
binary sort row therefore has seven fields instead of the original six, 
increasing its fixed part from 56 to 64 bytes per row, including for non-null 
inputs. The compact wire serializer avoids extra network bytes, but does not 
remove this local sorting cost. Please project back to the original three-field 
value schema before sorting so that the optimization does not reduce 
sort-buffer capacity unnecessarily. The resulting spill/runtime impact still 
needs measurement.
   
   Please add on/off coverage for both engines, checking equivalent results and 
index coverage and verifying that the disabled mode restores the original path. 
A Flink low-null case that spills would also help validate the second change. 
File-equivalence comparisons should continue to use identical sampled 
boundaries.
   
   The network reduction is useful, but the PR's local benchmark reports 94 -> 
162 ms at 80% nulls despite roughly halving shuffle bytes. This does not 
establish a cluster-wide slowdown, but it also does not establish that 80% is a 
suitable default activation threshold. A multi-worker benchmark covering CPU, 
network and spill costs would help decide when to enable it by default.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to