Zoltán Borók-Nagy created IMPALA-15119:
------------------------------------------
Summary: De-duplicate repeated VARIANT metadata StringValues
during row-batch serialization in KrpcDataStreamSender
Key: IMPALA-15119
URL: https://issues.apache.org/jira/browse/IMPALA-15119
Project: IMPALA
Issue Type: Sub-task
Reporter: Zoltán Borók-Nagy
A VARIANT value carries two inline StringValues — metadata and value (both
BINARY). The metadata (the field-name dictionary) is typically identical across
every variant in a column, so a batch of N variants serializes N identical
metadata blobs. We can serialize each distinct metadata blob once and have
subsequent identical slots reference it.
Why it's cheap on the wire: Serialization already rewrites each
StringValue::ptr into an offset into tuple_data_, and the receiver resolves it
back via ConvertOffsetsToPointers. Two slots pointing at the same offset share
the same bytes after deserialization, so dedup needs no wire-format or receiver
change, i.e. it's a sender-side decision to reuse a prior offset and skip the
copy. Benefit over existing LZ4: smaller uncompressed buffer, dedup across the
whole batch (beyond LZ4's ~64KB window), and a multiplied saving in broadcast
(one serialize → N receivers).
Scope / key risk: There are two independent serialization paths.
RowBatch::Serialize → DeepCopyVarlenData handles
UNPARTITIONED/RANDOM/single-channel; OutboundRowBatch::AppendRow →
TryDeepCopyStrings (codegen'd) handles HASH_PARTITIONED/KUDU.
A fix in only the first path gives zero benefit to hash-partitioned shuffles.
Dedup must be scoped to variant-metadata slots (per-slot flag) to avoid taxing
ordinary string workloads.
Proposed phasing:
# Path A (broadcast/unpartitioned) — content→offset dedup map threaded through
TotalByteSize/SerializeInternal, gated by a variant-metadata slot flag. Low
risk, no codegen.
# Path B (hash-partitioned) — same dedup in the codegen'd TryDeepCopyStrings
with a per-collector-batch map.
# (Stretch) Transmit metadata once per stream as a receiver-cached dictionary,
eliminating the redundancy entirely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
