[
https://issues.apache.org/jira/browse/IMPALA-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091975#comment-18091975
]
ASF subversion and git services commented on IMPALA-15118:
----------------------------------------------------------
Commit d9a3941b4ca9dc90611fff58d2d624b674cce807 in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d9a3941b4 ]
IMPALA-15118: Copy ByteBuffers for partitionList_
IcebergContentFileStore.fromThrift() didn't copy the contents
of ByteBuffers when filling partitionList_.
Thrift's binary handling in Java is special as the ByteBuffer
object in deserialized Thrift object can still reference the
transport buffer. Keeping the reference to that can almost
double the size of IcebergContentFileStore in the coordinator.
Another issue fixed is that partitionMap_ and partitionList_ did
not share the ByteBuffers after fromThrift()
Improvement for test table with 1M files, 25K partitions:
665MB->389MB
Change-Id: Ib39940dd64aadcb094b099e3863dd42c22afc2b8
Reviewed-on: http://gerrit.cloudera.org:8080/24515
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> partitionList_ references Thrift transport buffer
> -------------------------------------------------
>
> Key: IMPALA-15118
> URL: https://issues.apache.org/jira/browse/IMPALA-15118
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Csaba Ringhofer
> Priority: Critical
> Labels: impala-iceberg
>
> IMPALA-14564 deduplicated Iceberg partition info and individual files
> referred to their partition only with an integer.
> IcebergContentFileStore.partitionList_ is indexed with this integer to get
> actual partition info.
> The issues is that when filling partitionList_ in
> IcebergContentFileStore.fromThrift() the byteBuffers are not copied and can
> still reference the whole Thrift binary transport buffer (this issue is
> specific to binary type in Thrift in Java). This cam almost doubles the size
> of IcebergContentFileStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
