Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8825 )
Change subject: IMPALA-5990: End-to-end compression of metadata ...................................................................... IMPALA-5990: End-to-end compression of metadata Currently the catalog data is compressed in the statestore, but uncompressed when passed between FE and BE. It results in a ~2GB limit on the metadata. IMPALA-3499 introduced a workaround in the impalad but there isn't one in the catalogd. This patch aims to increase the size limit for statestore updates, reduce the copying of the metadata and reduce the memory footprint. With this patch, the catalog objects are passed and (de)compressed between FE and BE one at a time. The new limits are: - A single catalog object cannot be larger than ~2GB. - A statestore catalog update cannot be larger than ~4GB. It is compressed size if FLAGS_compact_catalog_topic is true. The behavior of the catalog op executer is not changed. The data is not compressed and the size limit is still 2GB. Testing: Ran existing tests. A test for compressing and decompressing catalog objects is added. Manually tested with a 1.95GB catalog object and a 3.90 GB uncompressed statestore update. Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Reviewed-on: http://gerrit.cloudera.org:8080/8825 Reviewed-by: Tianyi Wang <tw...@cloudera.com> Tested-by: Impala Public Jenkins --- M be/src/catalog/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h A be/src/catalog/catalog-util-test.cc M be/src/catalog/catalog-util.cc M be/src/catalog/catalog-util.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/rpc/thrift-util.h M be/src/service/fe-support.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/CatalogInternalService.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/TByteBuffer.java M testdata/workloads/functional-query/queries/QueryTest/exprs.test 27 files changed, 815 insertions(+), 563 deletions(-) Approvals: Tianyi Wang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Gerrit-Change-Number: 8825 Gerrit-PatchSet: 13 Gerrit-Owner: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>