GitHub user yuanchenl opened a pull request:
https://github.com/apache/incubator-quickstep/pull/338
Fixed the gRPC Problem for Data Exchange
Fixed the data exchange problem encountered in distributed quickstep.
It turned out that problem of data exchange is caused by the gRPC channel's
limit on the maximum receive size (4 MiB, 4194304 Bytes).
In DataExchange.proto
`message PullResponse {
bool is_valid = 1;
uint64 num_slots = 2;
bytes block = 3;
}`
the block size is 4MiB (which is fine), but there are 9 bytes of overhead
(num_slots 8 bytes and is_valid 1 byte). These extra bytes are the essential
reason that the data exchange failed.
In this commit, I set the limit of maximum receive size to 4194213 (4MiB +
9bytes), which is right enough for us.
With this commit, the distributed quickstep is able to pass ssb test with a
scale factor of 10
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yuanchenl/incubator-quickstep
data_exchange_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/338.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #338
----
commit a4d4c6f4397b14516471659ad510812c09863e8d
Author: Yuanchen Li <yli@...>
Date: 2018-03-13T19:58:00Z
Fixed the gRPC Problem for Data Exchange
----
---
