JingsongLi commented on PR #11076: URL: https://github.com/apache/paimon/pull/11076#issuecomment-6785720722
The explicit serializer fixes the Kryo registration issue; I rechecked the strict-registration case successfully on `8e646d5c4b`. Before enabling this optimization by default, I suggest two follow-ups: 1. **Add a shared Spark/Flink build option that restores the original execution path when disabled.** For example, `sorted-index.null-bitmap-shuffle.enabled` (proposed name): `true` permits the existing adaptive compression, while `false` restores Spark's original SQL `repartitionByRange` path and Flink's original three-field schema without the compression/expansion operators or custom transport serializer. Merely passing `compress=false` to `NullBitmapShuffle.sort` is insufficient because that still uses the new `sortUncompressed` implementation. The switch should provide a way to isolate and avoid regressions in the new shuffle implementation, as well as compression overhead. Given the current performance evidence, I would prefer an opt-in default initially. 2. **Remove the bitmap payload field before Flink's local sort.** `NullBitmapExpandOperator` currently emits a four-field value even after expansion, and `SortUtils` combines it with the three-field sort key. The binary sort row therefore has seven fields instead of the original six, increasing its fixed part from 56 to 64 bytes per row, including for non-null inputs. The compact wire serializer avoids extra network bytes, but does not remove this local sorting cost. Please project back to the original three-field value schema before sorting so that the optimization does not reduce sort-buffer capacity unnecessarily. The resulting spill/runtime impact still needs measurement. Please add on/off coverage for both engines, checking equivalent results and index coverage and verifying that the disabled mode restores the original path. A Flink low-null case that spills would also help validate the second change. File-equivalence comparisons should continue to use identical sampled boundaries. The network reduction is useful, but the PR's local benchmark reports 94 -> 162 ms at 80% nulls despite roughly halving shuffle bytes. This does not establish a cluster-wide slowdown, but it also does not establish that 80% is a suitable default activation threshold. A multi-worker benchmark covering CPU, network and spill costs would help decide when to enable it by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
