[
https://issues.apache.org/jira/browse/KUDU-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-3450:
--------------------------------
Description:
As one can see from the code in the
[SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110]
and in [MessageIO utility Java
class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73],
it's not possible to receive messages in PB format that are larger than a
pre-defined limit. Also, there isn't a way to pass
{{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data in
multiple chunks in the SubprocessProtocol by design.
The default setting for the maximum message size is 1MiByte in the
implementation of [the relevant
C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46]
and [Java subprocess
components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
The implementation of the related C++ components has the setting effectively
hard-coded and there isn't a way to change it. The implementation of the
related Java components has this setting configurable via the {{maxMsgBytes}}
configuration property.
Anyways, nether the code in {{RangerClient}} nor the code in
{{RangerProtocolHandler}} honors the limit:
*
[RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
*
[RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
*
[ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
With that, it's possible to get into an impasse situation from the both sides.
For example, if a large request has been generated by
{{RangerClient::AuthorizeActionMultipleColumns()}} or
{{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace
might be observed in the output of the subprocess that runs the Ranger client
plugin:
{noformat}
Exception in thread "main" java.util.concurrent.CompletionException:
org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf
message
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
the protobuf message
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
... 3 more
Caused by: java.io.IOException: message size (2763197) exceeds maximum message
size (1048576)
at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
... 4 more
{noformat}
was:
As one can see from the code in the
[SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110]
and in [MessageIO utility Java
class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73],
it's not possible to receive messages in PB format that are larger than a
pre-defined limit. Also, there isn't a way to pass
{{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data in
multiple chunks in the SubprocessProtocol by design.
The default setting for the maximum message size is as 1MiByte for the
implementation of [the relevant
C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46]
and [Java subprocess
components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
The implementation of the related C++ components has the setting effectively
hard-coded and there isn't a way to change it. The implementation of the
related Java components has this setting configurable via the {{maxMsgBytes}}
configuration property.
Anyways, nether the code in {{RangerClient}} nor the code in
{{RangerProtocolHandler}} honors that limit:
*
[RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
*
[RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
*
[ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
With that, it's possible to get into an impasse situation from the both sides.
For example, if a large request has been generated by
{{RangerClient::AuthorizeActionMultipleColumns()}} or
{{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace
might be observed in the output of the subprocess that runs the Ranger client
plugin:
{noformat}
Exception in thread "main" java.util.concurrent.CompletionException:
org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf
message
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
the protobuf message
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
... 3 more
Caused by: java.io.IOException: message size (2763197) exceeds maximum message
size (1048576)
at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
... 4 more
{noformat}
> SubprocessProtocol has a hard-coded limit on message size, but RangerClient
> doesn't honor that while generating requests
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3450
> URL: https://issues.apache.org/jira/browse/KUDU-3450
> Project: Kudu
> Issue Type: Bug
> Components: subprocess
> Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0
> Reporter: Alexey Serbin
> Priority: Major
>
> As one can see from the code in the
> [SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110]
> and in [MessageIO utility Java
> class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73],
> it's not possible to receive messages in PB format that are larger than a
> pre-defined limit. Also, there isn't a way to pass
> {{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data
> in multiple chunks in the SubprocessProtocol by design.
> The default setting for the maximum message size is 1MiByte in the
> implementation of [the relevant
> C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46]
> and [Java subprocess
> components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
> The implementation of the related C++ components has the setting effectively
> hard-coded and there isn't a way to change it. The implementation of the
> related Java components has this setting configurable via the {{maxMsgBytes}}
> configuration property.
> Anyways, nether the code in {{RangerClient}} nor the code in
> {{RangerProtocolHandler}} honors the limit:
> *
> [RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
> *
> [RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
> *
> [ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
> With that, it's possible to get into an impasse situation from the both
> sides. For example, if a large request has been generated by
> {{RangerClient::AuthorizeActionMultipleColumns()}} or
> {{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace
> might be observed in the output of the subprocess that runs the Ranger client
> plugin:
> {noformat}
> Exception in thread "main" java.util.concurrent.CompletionException:
> org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the
> protobuf message
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
> the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: message size (2763197) exceeds maximum
> message size (1048576)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
