[
https://issues.apache.org/jira/browse/IMPALA-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864745#comment-17864745
]
Ashwani Raina commented on IMPALA-13202:
----------------------------------------
[~stigahuang]
1. During one of the steps, you started a impalad cluster with some arguments:
bin/start-impala-cluster.py -r
--impalad_args="-kudu_mutation_buffer_size=56477399
-kudu_error_buffer_size=56477399 -rpc_max_message_size=2147483647"
While kudu_mutation_buffer_size and kudu_error_buffer_size seem to be impala
specific, what is the purpose of setting rpc_max_message_size to 2GB here? Are
you trying to change the rpc max message size limit on kudu server or client?
2. As you correctly mentioned, the limit is hidden under Kudu code. And the RPC
frame length limit error you are getting at the end is expected if the error
was hit at server side. To solve that, one can increase this limit on Kudu
tserver, by using Kudu CLI at runtime as follows:
{code:java}
kudu tserver set_flag <tablet_server> rpc_max_message_size <new_size>{code}
I am not sure how you can do that in your impala dev env. If the kudu tserver
runs as a separate binary in your env, you could try using above command and
then see if you get the error again.
3. Another possibility is that you are hitting the limit check at kudu client
side (probably is what you are hitting because meta_cache resides on client
side). In such a case, the server already has an increased limit (> 53477464)
but client side still has default limit of 52428800 bytes for a RPC message
which is why it cannot accommodate the result while receiving. For such a case
as well, there is a way in Kudu CLI to tell the client side to increase the
default size limit (check this [https://gerrit.cloudera.org/c/20535/)] For
example, in your case you are trying to scan the whole table at the end when
you get the rpc error. From kudu CLI, one could run
{code:java}
kudu table scan <master_address> <table_name>
--rpc_max_message_size=<new_val>{code}
to increase the receiving limit at client side.
TL;DR
If the rpc max size needs to be changed on kudu tablet server side, there is
already an option to do that at runtime (that can be run using set_flag option
in kudu CLI). I am not sure how you can do that from your impala dev env if you
are not using kudu CLI. It would depend on how kudu server runs in your impala
dev env.
If the rpc max size needs to be changed on kudu CLI tool at client side, there
is already an option to do that at runtime (that can be run setting the flag
itself from kudu CLI)
If the rpc max size needs to be changed on client side, I believe it needs to
be done from the user (impala in this case) of client library i.e.
libkudu_client.so. Just like kudu CLI tool does it. I don't fully understand
the way impala dev env runs kudu code but would it be possible to verify after
you change the default value of "rpc_max_message_size" inside
be/src/kudu/rpc/transfer.cc to 2GB ? I understand RpcMgr::Init function sets
the value to 2GB but I am not sure if that code path is executed in your case.
Maybe you have already verified that, just want to rule out such possibility.
> KRPC flags used by libkudu_client.so can't be configured
> --------------------------------------------------------
>
> Key: IMPALA-13202
> URL: https://issues.apache.org/jira/browse/IMPALA-13202
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Priority: Critical
> Attachments: data.parquet
>
>
> The way Impala integrates with KRPC is porting the KRPC codes into the Impala
> code base. Flags and methods of KRPC are defined as GLOBAL in the impalad
> executable. libkudu_client.so also compiles from the same KRPC codes and have
> duplicate flags and methods defined as HIDDEN.
> To be specifit, both the impalad executable and libkudu_client.so have the
> symbol for kudu::rpc::InboundTransfer::ReceiveBuffer()
> {noformat}
> $ readelf -s --wide be/build/latest/service/impalad | grep ReceiveBuffer
> 11118: 00000000022f5c88 1936 FUNC GLOBAL DEFAULT 13
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> 81380: 00000000022f5c88 1936 FUNC GLOBAL DEFAULT 13
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> $ readelf -s --wide
> toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so
> | grep ReceiveBuffer
> 1601: 0000000000086e4a 108 FUNC LOCAL DEFAULT 12
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE.cold
> 11905: 00000000001fec60 2076 FUNC LOCAL HIDDEN 12
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> $ c++filt
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> kudu::rpc::InboundTransfer::ReceiveBuffer(kudu::Socket*, kudu::faststring*)
> {noformat}
> KRPC flags like rpc_max_message_size are also defined in both the impalad
> executable and libkudu_client.so:
> {noformat}
> $ readelf -s --wide be/build/latest/service/impalad | grep
> FLAGS_rpc_max_message_size
> 14380: 0000000006006738 8 OBJECT GLOBAL DEFAULT 30
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> 80396: 0000000006006741 1 OBJECT GLOBAL DEFAULT 30
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 81399: 0000000006006741 1 OBJECT GLOBAL DEFAULT 30
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 117873: 0000000006006738 8 OBJECT GLOBAL DEFAULT 30
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> $ readelf -s --wide
> toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so
> | grep FLAGS_rpc_max_message_size
> 11882: 00000000008d61e1 1 OBJECT LOCAL HIDDEN 27
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 11906: 00000000008d61d8 8 OBJECT LOCAL DEFAULT 27
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> $ c++filt _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> fLI64::FLAGS_rpc_max_message_size {noformat}
> libkudu_client.so uses its own methods and flags. The flags are HIDDEN so
> can't be modified by Impala codes. E.g. IMPALA-4874 bumps
> FLAGS_rpc_max_message_size to 2GB in RpcMgr::Init(), but the HIDDEN variable
> FLAGS_rpc_max_message_size used in libkudu_client.so is still the default
> value 50MB (52428800). We've seen error messages like this in the master
> branch:
> {code:java}
> I0708 10:23:31.784974 2943 meta_cache.cc:294]
> c243bda4702a5ab9:0ba93d2400000001] tablet 0c8f3446538449ee9d3df5056afe775e:
> replica e0e1db54dab74f208e37ea1b975595e5 (127.0.0.1:31202) has failed:
> Network error: TS failed: RPC frame had a length of 53477464, but we only
> support messages up to 52428800 bytes long.{code}
> CC [~joemcdonnell] [~wzhou] [~aserbin]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
