[
https://issues.apache.org/jira/browse/KUDU-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Chennaka updated KUDU-3489:
------------------------------------
Description:
With KUDU-3450 we started supporting messages of size up to 8MB by default to
be transmitted between Kudu master and subprocess server. Given the the maximum
pipe buffer size is 1MB in linux machines [1] we see flaky failures if the size
of the message is more than 1MB between Kudu master and the subprocess server.
[1] [https://man7.org/linux/man-pages/man7/pipe.7.html] (pipe-max-size)
Example failure:
{code:java}
Exception in thread "main" java.util.concurrent.CompletionException:
org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf
message
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
the protobuf message
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
... 3 more
Caused by: java.io.IOException: unable to receive message, expected (2814368)
bytes but read (1777660) bytes
at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
... 4 more
{code}
was:
With KUDU-3450 we started supporting messages of size up to 8MB by default to
be transmitted between Kudu master and subprocess server. Given the the maximum
pipe buffer size is 1MB in linux machines [1] we see flaky failures if the size
of the message is more than 1MB between Kudu master and the subprocess server.
[1] [https://man7.org/linux/man-pages/man7/pipe.7.html] (pipe-max-size)
Example failure:
Exception in thread "main" java.util.concurrent.CompletionException:
org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf
message
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
the protobuf message
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
... 3 more
Caused by: java.io.IOException: unable to receive message, expected (2814368)
bytes but read (1777660) bytes
at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
... 4 more
> Subprocess communication with large messages
> --------------------------------------------
>
> Key: KUDU-3489
> URL: https://issues.apache.org/jira/browse/KUDU-3489
> Project: Kudu
> Issue Type: Bug
> Reporter: Abhishek Chennaka
> Priority: Major
>
> With KUDU-3450 we started supporting messages of size up to 8MB by default to
> be transmitted between Kudu master and subprocess server. Given the the
> maximum pipe buffer size is 1MB in linux machines [1] we see flaky failures
> if the size of the message is more than 1MB between Kudu master and the
> subprocess server.
> [1] [https://man7.org/linux/man-pages/man7/pipe.7.html] (pipe-max-size)
> Example failure:
> {code:java}
> Exception in thread "main" java.util.concurrent.CompletionException:
> org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the
> protobuf message
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
> the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: unable to receive message, expected (2814368)
> bytes but read (1777660) bytes
> at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
