[
https://issues.apache.org/jira/browse/IMPALA-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837712#comment-17837712
]
Quanlong Huang commented on IMPALA-12350:
-----------------------------------------
[~saulius.vl] Thanks for reporting this! There seems to be several issues.
The size of a catalog topic message (topic delta) will have a limit (configured
by thrift_rpc_max_message_size) after upgrading to thrift-0.16 since
Impala-4.2. When transfering the whole catalog topic to a newly added/restarted
coordinator, the topic message size could hit the limit.
{quote}Interestingly the catalog topic increased significantly after upgrading
from 3.4.0 to 4.2.0 - from ~800mb to ~3.4gb.
{quote}
In 4.2.0, catalogd sends catalog updates in partition level
(enable_incremental_metadata_updates=true). In 3.4.0, it sends them in table
level. So there are more topic entrics in 4.2.0. On the other hand. the
compression rate of catalog objects will be smaller in 4.2.0 since compressing
the whole table save more space than compressing partitions individually. We
recommend switching from the legacy catalog mode to the local catalog mode so
catalog objects sent to the catalog topics will be pretty small, which will
solve this issue.
To turn on local catalog mode, set use_local_catalog=true on all coordinators
and set catalog_topic_mode=minimal on catalogd.
Sorry for the late reply. Any feedbacks will be appreciated!
> Daemon fails to initialize large catalog
> ----------------------------------------
>
> Key: IMPALA-12350
> URL: https://issues.apache.org/jira/browse/IMPALA-12350
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.2.0
> Reporter: Saulius Valatka
> Priority: Major
>
> When the statestored catalog topic is large enough (>2gb) daemons fail to
> restart and get stuck in a loop:
> {{I0808 13:07:17.702653 3633556 Frontend.java:1618] Waiting for local catalog
> to be initialized, attempt: 2068}}
>
> The statestored reports errors as follows:
> {{I0808 13:07:05.587296 2134270 thrift-util.cc:196] TSocket::write_partial()
> send() <Host: gs1-hdp-data70 Port: 23000>: Broken pipe}}
> {{I0808 13:07:05.587356 2134270 client-cache.h:362] RPC Error: Client for
> gs1-hdp-data70:23000 hit an unexpected exception: write() send(): Broken
> pipe, type: N6apache6thrift9transport19TTransportExceptionE, rpc:
> N6impala20TUpdateStateResponseE, send: not done}}
> {{I0808 13:07:05.587365 2134270 client-cache.cc:174] Broken Connection,
> destroy client for gs1-hdp-data70:23000}}
>
> If this happens we are forced to restart statestore and thus the whole
> cluster, meaning that we can't tolerate failure from even a single daemon.
> Interestingly the catalog topic increased significantly after upgrading from
> 3.4.0 to 4.2.0 - from ~800mb to ~3.4gb. Invalidate/refresh operations also
> became significantly slower (~10ms -> 5s).
> Probably related to thrift_rpc_max_message_size? but I see the maximum value
> is 2gb.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
