[
https://issues.apache.org/jira/browse/IMPALA-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-12350.
-------------------------------------
Resolution: Workaround
Thank [~saulius.vl]! I'll resove this as workaround provided.
We will track the fix of sending a >2GB initial catalog update in IMPALA-13020.
Feel free to file new JRIAs for other issues.
BTW, the slower invalidate/refresh operations might be fixed by IMPALA-11501.
> Daemon fails to initialize large catalog
> ----------------------------------------
>
> Key: IMPALA-12350
> URL: https://issues.apache.org/jira/browse/IMPALA-12350
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.2.0
> Reporter: Saulius Valatka
> Priority: Major
>
> When the statestored catalog topic is large enough (>2gb) daemons fail to
> restart and get stuck in a loop:
> {{I0808 13:07:17.702653 3633556 Frontend.java:1618] Waiting for local catalog
> to be initialized, attempt: 2068}}
>
> The statestored reports errors as follows:
> {{I0808 13:07:05.587296 2134270 thrift-util.cc:196] TSocket::write_partial()
> send() <Host: gs1-hdp-data70 Port: 23000>: Broken pipe}}
> {{I0808 13:07:05.587356 2134270 client-cache.h:362] RPC Error: Client for
> gs1-hdp-data70:23000 hit an unexpected exception: write() send(): Broken
> pipe, type: N6apache6thrift9transport19TTransportExceptionE, rpc:
> N6impala20TUpdateStateResponseE, send: not done}}
> {{I0808 13:07:05.587365 2134270 client-cache.cc:174] Broken Connection,
> destroy client for gs1-hdp-data70:23000}}
>
> If this happens we are forced to restart statestore and thus the whole
> cluster, meaning that we can't tolerate failure from even a single daemon.
> Interestingly the catalog topic increased significantly after upgrading from
> 3.4.0 to 4.2.0 - from ~800mb to ~3.4gb. Invalidate/refresh operations also
> became significantly slower (~10ms -> 5s).
> Probably related to thrift_rpc_max_message_size? but I see the maximum value
> is 2gb.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
