Hi Ted, Really appreciate your prompt response and detailed explanation!
For some reason, ZK could be abused for writing large data objects. I understand we should correctly use ZK for coordination that ZK is best at. It's definitely something we could improve how we use ZK. But maybe it'd be a long run to arrive. Thanks for the clarification :) Back to the jute maxbuffer setting. With the consistent values 1 MB on both client and server, I am still able to produce it: request is sent to server as it throws IOException "Len error" and closes the connection. The client log is below, which does not give descriptive enough info like "Len error". [main-SendThread(localhost:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x1003e7613ab0005 for sever localhost:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. java.io.IOException: Connection reset by peer With this, can I assume the zk client does not fail the request? I also dig into the code, it seems the request reaches the server and the server fails the request. I am actually expecting the request can be failed earlier on the client side and then get descriptive info "the packet size is too large". Is this (when writing, client jute.maxbuffer is not honored) expected? I think if the client side fails the request and gives more descriptive info/specific exception, that'd be great and it's what I would expect. -Huizhi On Fri, Jan 8, 2021 at 12:01 AM Ted Dunning <ted.dunn...@gmail.com> wrote: > > Let's be clear from the start, storing large data objects in Zookeeper is > strongly discouraged. If you want to store large objects with good > consistency models, store the data in something else (like a distributed > file system or key value store), commit the data and then use ZK to provide > a reference to that data. Zookeeper is intended for coordination, not data > storage. It is not a reasonable alternative to a noSQL database. > > That said, good and informative error messages are always useful and are > better than anonymous errors. Even if the connection is closed related to > the error, it would be nice to give some decent feedback. > > On the other hand, the error that you are seeing sounds like your client > and your server have inconsistent settings for the maximum jute buffer > size. If the client has a larger setting than the server, then the server > will run out of buffer space before reading the entire request. To the > server, this will look like a network error and there is little that the > server can do to recover the connection safely because some bytes may > already have been lost due to the short read. As such closing the > connection is pretty much all that can be done. > > If buffer lengths on client and server match and the client tries a long > write, I believe the write will fail on the client side with a much more > descriptive message. > > One thing that could plausibly be done would be to enhance the initial > handshake between client and server so mismatch in buffer sizes are > detected more aggressively. Since a length could be exchanged in a fixed > size, this could be done while keeping the connection healthy which would, > in turn, allow a useful error message that drives to the true source of the > error (i.e. the configuration). This would, however, require a protocol > change which is always a sensitive change. > > Can you determine if the root cause of your problem is inconsistent > settings between client and server? > > > > On Thu, Jan 7, 2021 at 10:27 PM Huizhi Lu <h...@apache.org> wrote: > > > Hi ZK Experts, > > > > I would like to ask a quick question. As we know, assume we are using > > the default 1 MB jute.maxbuffer, if a zk client tries to write a large > > znode > 1MB, the server will fail it. Server will log "Len error" and > > close the connection. The client will receive a connection loss. In a > > third party ZkClient lib (eg. I0IZkClient), it'll keep retrying the > > operation upon connection loss. And this forever retrying might have a > > chance to take down the zk server. > > > > I believe the zk community must have considered such a situation. I > > wonder why zk server does not handle the error a bit better and send a > > clearer response to the client, eg. KeeperException.PacketLenError > > (and zk server does not really have to close the connection), so the > > client knows the error is non retryable. I think there must be some > > reasons I am not aware of that zk does not offer it, so I'd like to > > ask here. Or is there any ticket/email thread that has discussed this? > > > > Maybe zk would expect the app client to handle connection loss > > appropriately, eg. by having a retry strategy(backoff retry, limiting > > the retry count, etc.). Is this what zk would expect, instead of > > returning a PacketLenError exception? > > > > Really appreciate any input. > > > > > > Best, > > -Huizhi > >
