[
https://issues.apache.org/jira/browse/KUDU-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742859#comment-17742859
]
ASF subversion and git services commented on KUDU-3489:
-------------------------------------------------------
Commit 2e953e48f91b2d099f6c6f3f18921c74d2ecfeeb in kudu's branch
refs/heads/master from Abhishek Chennaka
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=2e953e48f ]
[subprocess] KUDU-3489: Support reading large messages through pipes
This patch enables the subprocess server to be able to read messages
larger than 1MB which was otherwise flaky by reading the input stream
messages until we encounter EOF. This issue is noticed when large sized
requests are made to the subprocess server and it fails in receiving
the complete messages.
In addition made a small log change to MessageIO.java to display the
exception message correctly.
Change-Id: I6523fdaaca19ee089dbac52a7dedec8847926a6c
Reviewed-on: http://gerrit.cloudera.org:8080/20180
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Abhishek Chennaka <[email protected]>
> Subprocess communication with large messages
> --------------------------------------------
>
> Key: KUDU-3489
> URL: https://issues.apache.org/jira/browse/KUDU-3489
> Project: Kudu
> Issue Type: Bug
> Reporter: Abhishek Chennaka
> Priority: Major
>
> With KUDU-3450 we started supporting messages of size up to 8MB by default to
> be transmitted between Kudu master and subprocess server. Given the the
> maximum pipe buffer size is 1MB in linux machines [1] we see flaky failures
> if the size of the message is more than 1MB between Kudu master and the
> subprocess server.
> [1] [https://man7.org/linux/man-pages/man7/pipe.7.html] (pipe-max-size)
> Example failure:
> {code:java}
> Exception in thread "main" java.util.concurrent.CompletionException:
> org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the
> protobuf message
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
> the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: unable to receive message, expected (2814368)
> bytes but read (1777660) bytes
> at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
