Vincent Tran created IMPALA-7034:
------------------------------------
Summary: Increase scalability of metadata handling
Key: IMPALA-7034
URL: https://issues.apache.org/jira/browse/IMPALA-7034
Project: IMPALA
Issue Type: Improvement
Affects Versions: Impala 2.13.0
Reporter: Vincent Tran
Currently the practical limit for catalog topic update is in the neighborhood
of 4GB - the fundamental limit of max thrift message size. This is an
architectural limitation and not a resource limitation
Larger enterprise clusters with high file counts can easily surpass this with
normal usage.
The high level ask here is for a more scalable implementation for metadata
handling. The amount metadata that a cluster can handle should be proportional
to the amount of hardware resource that a user is willing to allocate to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
