[ 
https://issues.apache.org/jira/browse/HIVE-30271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30271:
----------------------------------
    Description: 
HiveSplitGenerator keeps split payloads in the Tez AM until their total reaches
hive.tez.input.fs.serialization.threshold (256 MiB); every split after that is 
written to the staging file system as a
file of its own and read back by its task. IcebergSplit Java-serializes each 
task group as a separate stream, so every
split repeats the class descriptors and the schema and partition spec of its 
tasks: about 5.5 KB per split, of which
only 264-568 bytes are unique to it.

On a 1TRC table with ~49.6k splits the scan before the DML stays just under the 
threshold (258 MB, event stage 0.1 s),
while the scans after a V2 or V3 merge-on-read DML go over it (338 / 319 MB, of 
which 70 / 51 MB is written to the file
system), and split generation grows from 3.2 s to 16.6 / 11.1 s.

Fix: IcebergSplit compresses its payload with the codec named by the new 
hive.iceberg.split.compression.codec
(snappy by default, zstd or none). The payload starts with the codec id, so the 
reader needs no configuration.
Measured per split: snappy 1.75-1.9x smaller (12x for splits carrying many 
partition-scoped delete files) at ~5 us of
AM time per split and no read-side cost.

>  Iceberg: Compress split payloads so delete-heavy scans stay under the split 
> serialization threshold
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30271
>                 URL: https://issues.apache.org/jira/browse/HIVE-30271
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Denys Kuzmenko
>            Priority: Major
>
> HiveSplitGenerator keeps split payloads in the Tez AM until their total 
> reaches
> hive.tez.input.fs.serialization.threshold (256 MiB); every split after that 
> is written to the staging file system as a
> file of its own and read back by its task. IcebergSplit Java-serializes each 
> task group as a separate stream, so every
> split repeats the class descriptors and the schema and partition spec of its 
> tasks: about 5.5 KB per split, of which
> only 264-568 bytes are unique to it.
> On a 1TRC table with ~49.6k splits the scan before the DML stays just under 
> the threshold (258 MB, event stage 0.1 s),
> while the scans after a V2 or V3 merge-on-read DML go over it (338 / 319 MB, 
> of which 70 / 51 MB is written to the file
> system), and split generation grows from 3.2 s to 16.6 / 11.1 s.
> Fix: IcebergSplit compresses its payload with the codec named by the new 
> hive.iceberg.split.compression.codec
> (snappy by default, zstd or none). The payload starts with the codec id, so 
> the reader needs no configuration.
> Measured per split: snappy 1.75-1.9x smaller (12x for splits carrying many 
> partition-scoped delete files) at ~5 us of
> AM time per split and no read-side cost.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to