[
https://issues.apache.org/jira/browse/IMPALA-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863983#comment-17863983
]
Quanlong Huang commented on IMPALA-13202:
-----------------------------------------
YoungJi Nam figured out a way to reproduce the issue by using the attached
[^data.parquet]. We need the following Kudu configs:
{code:java}
--unlock_unsafe_flags=true
--max_cell_size_bytes=1073741824
--max_cfile_block_size=1073741824{code}
In the Impala dev env, add them in
testdata/cluster/cdh7/node-*/etc/kudu/tserver.conf
Then start Impala cluster with the following configs:
{code:java}
-kudu_mutation_buffer_size=56477399
-kudu_error_buffer_size=56477399
-rpc_max_message_size=2147483647{code}
In the Impala dev env, the command is
{code:java}
bin/start-impala-cluster.py -r
--impalad_args="-kudu_mutation_buffer_size=56477399
-kudu_error_buffer_size=56477399 -rpc_max_message_size=2147483647"{code}
Create a Parquet table and a Kudu table
{code:sql}
create external table test_parquet (str string) stored as parquet;
create table test_kudu_large (
id int,
str string,
primary key(id)
) stored as kudu;{code}
Put the file [^data.parquet] into the location of test_parquet table and
REFRESH the table. Then INSERT the value into the Kudu table.
{code:sql}
insert into test_kudu_large select 1, str from test_parquet;{code}
Run a SELECT query on the Kudu table can see the error messages in impalad logs.
{code:sql}
select * from test_kudu_large;{code}
> KRPC flags used by libkudu_client.so can't be configured
> --------------------------------------------------------
>
> Key: IMPALA-13202
> URL: https://issues.apache.org/jira/browse/IMPALA-13202
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Priority: Critical
> Attachments: data.parquet
>
>
> The way Impala integrates with KRPC is porting the KRPC codes into the Impala
> code base. Flags and methods of KRPC are defined as GLOBAL in the impalad
> executable. libkudu_client.so also compiles from the same KRPC codes and have
> duplicate flags and methods defined as HIDDEN.
> To be specifit, both the impalad executable and libkudu_client.so have the
> symbol for kudu::rpc::InboundTransfer::ReceiveBuffer()
> {noformat}
> $ readelf -s --wide be/build/latest/service/impalad | grep ReceiveBuffer
> 11118: 00000000022f5c88 1936 FUNC GLOBAL DEFAULT 13
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> 81380: 00000000022f5c88 1936 FUNC GLOBAL DEFAULT 13
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> $ readelf -s --wide
> toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so
> | grep ReceiveBuffer
> 1601: 0000000000086e4a 108 FUNC LOCAL DEFAULT 12
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE.cold
> 11905: 00000000001fec60 2076 FUNC LOCAL HIDDEN 12
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> $ c++filt
> _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
> kudu::rpc::InboundTransfer::ReceiveBuffer(kudu::Socket*, kudu::faststring*)
> {noformat}
> KRPC flags like rpc_max_message_size are also defined in both the impalad
> executable and libkudu_client.so:
> {noformat}
> $ readelf -s --wide be/build/latest/service/impalad | grep
> FLAGS_rpc_max_message_size
> 14380: 0000000006006738 8 OBJECT GLOBAL DEFAULT 30
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> 80396: 0000000006006741 1 OBJECT GLOBAL DEFAULT 30
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 81399: 0000000006006741 1 OBJECT GLOBAL DEFAULT 30
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 117873: 0000000006006738 8 OBJECT GLOBAL DEFAULT 30
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> $ readelf -s --wide
> toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so
> | grep FLAGS_rpc_max_message_size
> 11882: 00000000008d61e1 1 OBJECT LOCAL HIDDEN 27
> _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
> 11906: 00000000008d61d8 8 OBJECT LOCAL DEFAULT 27
> _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> $ c++filt _ZN5fLI6426FLAGS_rpc_max_message_sizeE
> fLI64::FLAGS_rpc_max_message_size {noformat}
> libkudu_client.so uses its own methods and flags. The flags are HIDDEN so
> can't be modified by Impala codes. E.g. IMPALA-4874 bumps
> FLAGS_rpc_max_message_size to 2GB in RpcMgr::Init(), but the HIDDEN variable
> FLAGS_rpc_max_message_size used in libkudu_client.so is still the default
> value 50MB (52428800). We've seen error messages like this in the master
> branch:
> {code:java}
> I0708 10:23:31.784974 2943 meta_cache.cc:294]
> c243bda4702a5ab9:0ba93d2400000001] tablet 0c8f3446538449ee9d3df5056afe775e:
> replica e0e1db54dab74f208e37ea1b975595e5 (127.0.0.1:31202) has failed:
> Network error: TS failed: RPC frame had a length of 53477464, but we only
> support messages up to 52428800 bytes long.{code}
> CC [~joemcdonnell] [~wzhou] [~aserbin]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
