Alexey Serbin created KUDU-3450:
-----------------------------------
Summary: SubprocessProtocol has a hard-coded limit on message
size, but RangerClient doesn't honor that while generating requests
Key: KUDU-3450
URL: https://issues.apache.org/jira/browse/KUDU-3450
Project: Kudu
Issue Type: Bug
Components: subprocess
Reporter: Alexey Serbin
As one can see from the code in the
[SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110]
and in [MessageIO utility Java
class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73],
it's not possible to receive messages in PB format that are larger than a
pre-defined limit. Also, there isn't a way to pass
{{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data in
multiple chunks in the SubprocessProtocol by design.
The default setting for the maximum message size is as 1MiByte for the
implementation of [the relevant
C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46]
and [Java subprocess
components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
The implementation of the related C++ components has the setting effectively
hard-coded and there isn't a way to change it. The implementation of the
related Java components has this setting configurable via the {{maxMsgBytes}}
configuration property.
Anyways, nether the code in {{RangerClient}} nor the code in
{{RangerProtocolHandler}} honors that limit:
*
[RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
*
[RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
*
[ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
With that, it's possible to get into an impasse situation from the both sides.
For example, if a large request has been generated by
{{RangerClient::AuthorizeActionMultipleColumns()}} or
{{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace
might be observed in the output of the subprocess that runs the Ranger client
plugin:
{noformat}
Exception in thread "main" java.util.concurrent.CompletionException:
org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf
message
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
the protobuf message
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
... 3 more
Caused by: java.io.IOException: message size (2763197) exceeds maximum message
size (1048576)
at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
... 4 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
