zghong commented on PR #66477: URL: https://github.com/apache/doris/pull/66477#issuecomment-6332291905
Thanks @HappenLee — both points are addressed, and we also follow the latest upstream feature parity and harden identity distribution: > Duplicate `_initialize_hash_vals()` override Fixed in`ee14e2ed18d`. The override is simply removed; `IdentityHashPartitioner` now inherits the identical base implementation (`Crc32HashPartitioner<ShuffleChannelIds>::_initialize_hash_vals()`, same resize + zero-fill). No behavior change, one implementation left to maintain. > Per-row virtual dispatch Fixed in `6746a5efcd1`. Following the CRC32 pattern (`update_crcs_with_value`), a batch interface `IColumn::update_identity_hash()` was added and implemented by `ColumnVector`/`ColumnStr`/`ColumnNullable`/`ColumnDecimal` iterating their underlying storage directly. `_do_hash` collapses to a single virtual call, moving dispatch out of the row loop. Byte semantics are owned by the shared `RawValue::append_identity_bytes`, so batch and scalar paths (tablet routing, bucket pruning) are bit-identical. Subsequent commits fall into two blocks: - **End-to-end hardening of identity distribution**: planner/Nereids propagates the hash type across the full plan and rejects mixed-hash fragments instead of silent CRC32 fallback, runtime-filter pruning and IP/UUID/TIMESTAMP_NS identity bytes align with BE storage, metadata/restore validation includes the hash type, `identity_hash_internal` input validation is hardened, and BE UT + regression coverage is added throughout. - **Upstream feature parity**: `d4547e662d4` and `43f3edb7d15` follow upstream work to align with CRC32 and bucket query optimization — adding VARBINARY batch identity hash at the same interface level as CRC32, and grouping scan-key bucket prune ranges with the table's hash type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
