Hi Jens, The situation came about due to the race condition above. My client made two requests one after the other, which were dispatched immediately in a thread pool on the server. Since I'm on a multicore system, the two responses were written to the wire at exactly the same time. The first thing written to the wire is the int32 VERSION_1 (0x80010000). My client ended up receiving two of these concurrently.
You can probably imagine the rest yourself, but as such, when the client receives this, we go into TBinaryProtocol::readMessageBegin. It reads off the version as VERSION_1, fine. This method then calls readString(), which does a readI32 expecting the length of the string. However, the server has written another VERSION_1 to the wire, which then gets interpreted as the length of the string, and since it's greater than 0x7FFFFFFF, it's interpreted as being negative and the bytes "remaining" in the buffer is therefore larger. A fun one indeed! Andrew On 11 August 2014 12:17, Jens Geyer <[email protected]> wrote: > Hi Andrew, > > great to hear the good news! > > if you send readString a negative size on a TSocket, >> it will throw a string-out-of-bounds exception since >> [...] >> I don't know if that's a bug worth fixing [...] >> > > IMHO a negative size is not, well, a common use case. If I don't overlook > anything, the only place where this may happen is when the length exceeds > 0x7FFFFFFF, then getting casted into a signed int. So (again IMHO) this > looks not like a real bug to me. I just wonder how you did run into that > scenario? > > > Have fun, > JensG > > > > -----Ursprüngliche Nachricht----- From: Andrew Dodd > Sent: Saturday, August 9, 2014 8:30 PM > To: [email protected] > Subject: Re: Asynchronous RPC > > > Hi Jens, > > Thanks for the reply - I'd actually forgotten about this issue for a while. > I did some digging tonight, and the problem is that the server was sending > two messages at once on the same client, which breaks horribly. If I > synchronize on the client, then the problem goes away. > > Cheers, > Andrew > > PS. One slightly strange thing that happens is in TBinaryProtocol.java - if > you send readString a negative size on a TSocket, it will throw a > string-out-of-bounds exception since getBytesRemainingInBuffer (which would > normally catch this) is returning -1. I don't know if that's a bug worth > fixing or not, since my real-world problem was elsewhere. > > > > On 9 August 2014 09:51, Jens Geyer <[email protected]> wrote: > > Hi, >> >> >> My thought on this is that the one-way message isn't actually >> >>> completely one-way; that we do send some sort of acknowledgement >>> packet, and this is causing a mess on the receiving end. I also notice >>> that if I put a breakpoint inside my call-back to the client (inside the >>> server's code) and let it run a while after, that I don't get an error >>> and >>> the message does get received on the client, further strengthening >>> my suspicions. >>> >>> Can anyone confirm that this could be the issue? >>> >>> >> If you have a test case and the issue is reproducible with current trunk, >> you should file a JIRA ticket. >> >> And if (by accident :-) you already have a patch/pull request, >> see http://thrift.apache.org/docs/HowToContribute >> >> Have fun, >> JensG >> >> >> -----Ursprüngliche Nachricht----- From: Andrew Dodd >> Sent: Monday, July 21, 2014 7:18 AM >> To: [email protected] >> Subject: Asynchronous RPC >> >> >> Morning, >> >> Over the last few days I've been experimenting with the code provided by >> this article: >> http://joelpm.com/2009/04/03/thrift-bidirectional-async-rpc.html . >> However, >> that code doesn't exactly fit my needs. I checked out the source from >> GitHub and did the following things: >> >> 1. Updated it to work with Thrift 0.9.1 - everything is OK here. >> 2. Changed the Service to use a different callback method (it does mean >> several no-op implementations, but that is fine). >> 3. Changed the MessageDistributor to be instantiated once per client. This >> means that rather than sending one response to each client, it is only >> sent >> to the client which sent the request. >> >> Up to this point everything works fine. However, I don't really want one >> thread per client as it's potentially wasteful. So rather than do that, I >> made the MessageDistributor submit a runnable into a shared Executor >> Service. At this point, problems start to occur. When the client receives >> it the first response, I get this message: >> >> Exception in thread "Thread-2" java.lang.StringIndexOutOfBoundsException: >> String index out of range: -2147418111 >> at java.lang.String.checkBounds(String.java:371) >> at java.lang.String.<init>(String.java:415) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readString( >> TBinaryProtocol.java:326) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin( >> TBinaryProtocol.java:197) >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) >> at >> com.joelpm.bidiMessages.client.MessageReceiver.run( >> MessageReceiver.java:31) >> at java.lang.Thread.run(Thread.java:745) >> >> My thought on this is that the one-way message isn't actually completely >> one-way; that we do send some sort of acknowledgement packet, and this is >> causing a mess on the receiving end. I also notice that if I put a >> breakpoint inside my call-back to the client (inside the server's code) >> and >> let it run a while after, that I don't get an error and the message does >> get received on the client, further strengthening my suspicions. >> >> Can anyone confirm that this could be the issue? >> >> Thanks, >> Andrew >> >> >
