[ 
https://issues.apache.org/jira/browse/HIVE-30271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-30271:
----------------------------------
    Labels: pull-request-available  (was: )

>  Iceberg: Compress split payloads so delete-heavy scans stay under the split 
> serialization threshold
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30271
>                 URL: https://issues.apache.org/jira/browse/HIVE-30271
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Denys Kuzmenko
>            Priority: Major
>              Labels: pull-request-available
>
> HiveSplitGenerator keeps split payloads in the Tez AM until their total 
> reaches
> hive.tez.input.fs.serialization.threshold (256 MiB); every split after that 
> is written to the staging file system as a
> file of its own and read back by its task. IcebergSplit Java-serializes each 
> task group as a separate stream, so every
> split repeats the class descriptors and the schema and partition spec of its 
> tasks: about 5.5 KB per split, of which
> only 264-568 bytes are unique to it.
> On a 1TRC table with ~49.6k splits the scan before the DML stays just under 
> the threshold (258 MB, event stage 0.1 s),
> while the scans after a V2 or V3 merge-on-read DML go over it (338 / 319 MB, 
> of which 70 / 51 MB is written to the file
> system), and split generation grows from 3.2 s to 16.6 / 11.1 s.
> Fix: IcebergSplit compresses its payload with the codec named by the new 
> hive.iceberg.split.compression.codec
> (snappy by default, zstd or none). The payload starts with the codec id, so 
> the reader needs no configuration.
> Measured per split: snappy 1.75-1.9x smaller (12x for splits carrying many 
> partition-scoped delete files) at ~5 us of
> AM time per split and no read-side cost.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to