zhangqianqiong has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21845 )
Change subject: IMPALA-13385: Compact the response of ddl operations ...................................................................... IMPALA-13385: Compact the response of ddl operations At present, when catalogd responds to DDL operations, it sends the entire table object. This can lead to a massive transfer of table catalog when dealing with the hive partitioned table. In one of our customer's clusters, there is a hive partitioned table with over 4,000 columns, more than 20,000 partitions, and involving over 10 million hdfs files. When executing an `ALTER TABLE ADD PARTITION` operation on this table, the catalog being serialized for the table exceeds the java array size limit, resulting in the following exception: `java.long.OutOfMemoryError: Requested array size exceeds VM limit`. To alleviate the issue, we can use TCompactProtocol instead of TBinaryProtocol for thrift serialization. In an experiment with a hive table containing 160 partitions, I observed that using TCompactProtocol can reduce the serialized data size by 34.4% compared to the previous method. Change-Id: Idea9313c7f1f1596f3620e60b08a99efc7fa0466 --- M be/src/catalog/catalog.cc M be/src/rpc/jni-thrift-util.h M fe/src/main/java/org/apache/impala/service/JniCatalog.java 3 files changed, 33 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/21845/5 -- To view, visit http://gerrit.cloudera.org:8080/21845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idea9313c7f1f1596f3620e60b08a99efc7fa0466 Gerrit-Change-Number: 21845 Gerrit-PatchSet: 5 Gerrit-Owner: zhangqianqiong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
