Joe McDonnell created IMPALA-13020:
--------------------------------------
Summary: catalog-topic updates >2GB do not work due to Thrift's
max message size
Key: IMPALA-13020
URL: https://issues.apache.org/jira/browse/IMPALA-13020
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.4.0
Reporter: Joe McDonnell
Thrift 0.16.0 added a max message size to protect against malicious packets
that can consume a large amount of memory on the receiver side. This max
message size is a signed 32-bit integer, so it maxes out at 2GB (which we set
via thrift_rpc_max_message_size).
In catalog v1, the catalog-update statestore topic can become larger than 2GB
when there are a large number of tables / partitions / files. If this happens
and an Impala coordinator needs to start up (or needs a full topic update for
any other reason), it is expecting the statestore to send it the full topic
update, but the coordinator actually can't process the message. The
deserialization of the message hits the 2GB max message size limit and fails.
On the statestore side, it shows this message:
{noformat}
I0418 16:54:51.727290 3844140 statestore.cc:507] Preparing initial
catalog-update topic update for [email protected]:27000.
Size = 2.27 GB
I0418 16:54:53.889446 3844140 thrift-util.cc:198] TSocket::write_partial()
send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
I0418 16:54:53.889488 3844140 client-cache.cc:82] ReopenClient(): re-creating
client for mcdonnellthrift.vpc.cloudera.com:23000
I0418 16:54:53.889493 3844140 thrift-util.cc:198] TSocket::write_partial()
send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
I0418 16:54:53.889503 3844140 thrift-client.cc:116] Error closing connection
to: mcdonnellthrift.vpc.cloudera.com:23000, ignoring (write() send(): Broken
pipe)
I0418 16:54:56.052882 3844140 thrift-util.cc:198] TSocket::write_partial()
send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
I0418 16:54:56.052932 3844140 client-cache.h:363] RPC Error: Client for
mcdonnellthrift.vpc.cloudera.com:23000 hit an unexpected exception: write()
send(): Broken pipe, type: N6apache6thrift9transport19TTransportExceptionE,
rpc: N6impala20TUpdateStateResponseE, send: not done
I0418 16:54:56.052937 3844140 client-cache.cc:174] Broken Connection, destroy
client for mcdonnellthrift.vpc.cloudera.com:23000{noformat}
On the Impala side, it doesn't give a good error, but we see this:
{noformat}
I0418 16:54:53.889683 3214537 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 49632>
I0418 16:54:54.080694 3214136 Frontend.java:1837] Waiting for local catalog to
be initialized, attempt: 110
I0418 16:54:56.080920 3214136 Frontend.java:1837] Waiting for local catalog to
be initialized, attempt: 111
I0418 16:54:58.081131 3214136 Frontend.java:1837] Waiting for local catalog to
be initialized, attempt: 112
I0418 16:55:00.081358 3214136 Frontend.java:1837] Waiting for local catalog to
be initialized, attempt: 113{noformat}
With a patch Thrift that allows an int64_t max message size and setting that to
a larger value, the Impala was able to start up (even without restarting the
statestored).
Some clusters that upgrade to a newer version may hit this, as Thrift didn't
use to enforce this limit, so this is something we should fix to avoid upgrade
issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
