[
https://issues.apache.org/jira/browse/KUDU-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744391#comment-17744391
]
ASF subversion and git services commented on KUDU-3489:
-------------------------------------------------------
Commit 0b7e45732b64f2edf86dc4b3b0d6bf7b26228bd5 in kudu's branch
refs/heads/branch-1.17.x from Abhishek Chennaka
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=0b7e45732 ]
[subprocess] KUDU-3489: Support reading large messages through pipes
This patch enables the subprocess server to be able to read messages
larger than 1MB which was otherwise flaky by reading the input stream
messages until we encounter EOF. This issue is noticed when large sized
requests are made to the subprocess server and it fails in receiving
the complete messages.
In addition made a small log change to MessageIO.java to display the
exception message correctly.
Change-Id: I6523fdaaca19ee089dbac52a7dedec8847926a6c
Reviewed-on: http://gerrit.cloudera.org:8080/20180
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Abhishek Chennaka <[email protected]>
(cherry picked from commit 2e953e48f91b2d099f6c6f3f18921c74d2ecfeeb)
Reviewed-on: http://gerrit.cloudera.org:8080/20193
Tested-by: Yingchun Lai <[email protected]>
> Subprocess communication with large messages
> --------------------------------------------
>
> Key: KUDU-3489
> URL: https://issues.apache.org/jira/browse/KUDU-3489
> Project: Kudu
> Issue Type: Bug
> Reporter: Abhishek Chennaka
> Priority: Major
> Fix For: 1.17.0
>
>
> With KUDU-3450 we started supporting messages of size up to 8MB by default to
> be transmitted between Kudu master and subprocess server. Given the the
> maximum pipe buffer size is 1MB in linux machines [1] we see flaky failures
> if the size of the message is more than 1MB between Kudu master and the
> subprocess server.
> [1] [https://man7.org/linux/man-pages/man7/pipe.7.html] (pipe-max-size)
> Example failure:
> {code:java}
> Exception in thread "main" java.util.concurrent.CompletionException:
> org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the
> protobuf message
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
> the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: unable to receive message, expected (2814368)
> bytes but read (1777660) bytes
> at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
