[
https://issues.apache.org/jira/browse/IMPALA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847419#comment-17847419
]
ASF subversion and git services commented on IMPALA-13020:
----------------------------------------------------------
Commit 13df8239d82a61afc3196295a7878ca2ffe91873 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=13df8239d ]
IMPALA-13020 (part 1): Change thrift_rpc_max_message_size to int64_t
Thrift 0.16.0 introduced a max message size to protect
receivers against a malicious message allocating large
amounts of memory. That limit is a 32-bit signed integer,
so the max value is 2GB. Impala introduced the
thrift_rpc_max_message_size startup option to set that
for Impala's thrift servers.
There are times when Impala wants to send a message that
is larger than 2GB. In particular, the catalog-update
topic for the statestore can exceed 2GBs when there is
a lot of metadata loaded using the old v1 catalog. When
there is a 2GB max message size, the statestore can create
and send a >2GB message, but the impalads will reject
it. This can lead to impalads having stale metadata.
This switches to a patched Thrift that uses an int64_t
for the max message size for C++ code. It does not modify
the limit.
The MaxMessageSize error was being swallowed in TAcceptQueueServer.cpp,
so this fixes that location to always print MaxMessageSize
exceptions.
This is only patching the Thrift C++ library. It does not
patch the Thrift Java library. There are a few reasons for
that:
- This specific issue involves C++ to C++ communication and
will be solved by patching the C++ library.
- C++ is easy to patch as it is built via the native-toolchain.
There is no corresponding mechanism for patching our Java
dependencies (though one could be developed).
- Java modifications have implications for other dependencies
like Hive which use Thrift to communicate with HMS.
For the Java code that uses max message size, this converts
the 64-bit value to 32-bit value by capping the value at
Integer.MAX_VALUE.
Testing:
- Added enough tables to produce a >2GB catalog-topic and
restarted an impalad with a higher limit specific. Without
the patch, the catalog-topic update would be rejected by the
impalad. With the patch, it succeeds.
Change-Id: I681b1849cc565dcb25de8c070c18776ce69cbb87
Reviewed-on: http://gerrit.cloudera.org:8080/21367
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> catalog-topic updates >2GB do not work due to Thrift's max message size
> -----------------------------------------------------------------------
>
> Key: IMPALA-13020
> URL: https://issues.apache.org/jira/browse/IMPALA-13020
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0, Impala 4.3.0
> Reporter: Joe McDonnell
> Priority: Critical
>
> Thrift 0.16.0 added a max message size to protect against malicious packets
> that can consume a large amount of memory on the receiver side. This max
> message size is a signed 32-bit integer, so it maxes out at 2GB (which we set
> via thrift_rpc_max_message_size).
> In catalog v1, the catalog-update statestore topic can become larger than 2GB
> when there are a large number of tables / partitions / files. If this happens
> and an Impala coordinator needs to start up (or needs a full topic update for
> any other reason), it is expecting the statestore to send it the full topic
> update, but the coordinator actually can't process the message. The
> deserialization of the message hits the 2GB max message size limit and fails.
> On the statestore side, it shows this message:
> {noformat}
> I0418 16:54:51.727290 3844140 statestore.cc:507] Preparing initial
> catalog-update topic update for
> [email protected]:27000. Size = 2.27 GB
> I0418 16:54:53.889446 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:53.889488 3844140 client-cache.cc:82] ReopenClient(): re-creating
> client for mcdonnellthrift.vpc.cloudera.com:23000
> I0418 16:54:53.889493 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:53.889503 3844140 thrift-client.cc:116] Error closing connection
> to: mcdonnellthrift.vpc.cloudera.com:23000, ignoring (write() send(): Broken
> pipe)
> I0418 16:54:56.052882 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:56.052932 3844140 client-cache.h:363] RPC Error: Client for
> mcdonnellthrift.vpc.cloudera.com:23000 hit an unexpected exception: write()
> send(): Broken pipe, type: N6apache6thrift9transport19TTransportExceptionE,
> rpc: N6impala20TUpdateStateResponseE, send: not done
> I0418 16:54:56.052937 3844140 client-cache.cc:174] Broken Connection, destroy
> client for mcdonnellthrift.vpc.cloudera.com:23000{noformat}
> On the Impala side, it doesn't give a good error, but we see this:
> {noformat}
> I0418 16:54:53.889683 3214537 TAcceptQueueServer.cpp:355] New connection to
> server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 49632>
> I0418 16:54:54.080694 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 110
> I0418 16:54:56.080920 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 111
> I0418 16:54:58.081131 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 112
> I0418 16:55:00.081358 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 113{noformat}
> With a patch Thrift that allows an int64_t max message size and setting that
> to a larger value, the Impala was able to start up (even without restarting
> the statestored).
> Some clusters that upgrade to a newer version may hit this, as Thrift didn't
> use to enforce this limit, so this is something we should fix to avoid
> upgrade issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
