[
https://issues.apache.org/jira/browse/THRIFT-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649476#action_12649476
]
Bryan Duxbury commented on THRIFT-200:
--------------------------------------
So, it sounds like this issue is really about the Ruby libs doing the right
thing when attempting to serialize weird stuff. I think at least it should
close the connection. This would keep the server from mixing data from two
different invocations.
> arguments corrupted between ruby and java under concurrent load
> ---------------------------------------------------------------
>
> Key: THRIFT-200
> URL: https://issues.apache.org/jira/browse/THRIFT-200
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java), Library (Ruby)
> Environment: latest snapshot of thrift git HEAD
> running debian etch
> client is running ruby 1.8.5 (2006-08-25) [x86_64-linux]
> server is running java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
> Reporter: Eric Jensen
> Attachments: thrift_tcpdump.out
>
>
> We have a ruby client talking to a java server and are seeing falsely huge or
> negative string/array sizes causing Exceptions in the java server when it is
> under load. We can't reproduce this problem when querying it in serial, but
> with about 10 requests per second we get this corruption of our args. Many
> traces are below and I will attach a tcpdump. Any help would be
> appreciated...we're reasonably stuck now.
> our thrift interface is below. (we have had the same problems with and
> without the "async", although the trace we provide includes it.) we are
> calling the search method, then if that doesn't find the required results
> calling populate. we do set a timeout of 200ms on the sockets for both the
> server and client. we could pass empty arrays and some nil values, but never
> nil arrays or entries in those arrays. we have about 30 clients talking to
> the service during these errors.
> struct View {
> 1: bool complete,
> 2: list<i32> view = [],
> 3: i32 count = -1,
> }
> service Cache {
> View search( 1:list<list<string>> query,
> 2:i32 requested,
> 3:i32 offset,
> 4:i32 sinceId,
> 5:i32 maxId,
> 6:bool count )
>
> async void populate( 1:list<list<string>> query,
> 2:list<i32> view,
> 3:i32 completeSinceId,
> 4:i32 maxIndexedId )
> }
> our client code is basically
> Thrift::BinaryProtocol.new(Thrift::FramedTransport.new(Thrift::Socket.new('www4',
> 9090, 200)))
> between 4:35:01 and 04 we get
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap
> space
> at java.util.ArrayList.<init>(ArrayList.java:112)
> at Cache$search_args.read(Cache.java:314)
> at Cache$Processor$search.process(Cache.java:150)
> at Cache$Processor.process(Cache.java:142)
> at
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown
> Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> which on the client side appears as:
> Thrift::Exception: Socket: Timed out reading 4 bytes from www4:9090
> we've also gotten another OOM, although it is not in the attached tcpdump:
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap
> space
> at
> com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown
> Source)
> at Cache$search_args.read(Cache.java:344)
> at Cache$Processor$search.process(Cache.java:165)
> at Cache$Processor.process(Cache.java:157)
> at
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown
> Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> and later we get two of these in the same second:
> Nov 11, 2008 4:35:32 PM
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer invoke
> SEVERE: Unexpected exception while invoking!
> java.lang.NegativeArraySizeException
> at
> com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown
> Source)
> at Cache$search_args.read(Cache.java:324)
> at Cache$Processor$search.process(Cache.java:150)
> at Cache$Processor.process(Cache.java:142)
> at
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown
> Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Nov 11, 2008 4:35:32 PM
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer invoke
> SEVERE: Unexpected exception while invoking!
> java.lang.NegativeArraySizeException
> at
> com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown
> Source)
> at Cache$populate_args.read(Cache.java:707)
> at Cache$Processor$populate.process(Cache.java:167)
> at Cache$Processor.process(Cache.java:142)
> at
> com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown
> Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> which on the client side results in:
> Thrift::Exception: end of file reached
> We've also seen the NegativeArraySizeException in readStringBody from our
> populate method.
> We also tried TThreadPoolServer with the following client code and then got
> the same problem (with different trace of course):
> Thrift::BinaryProtocol.new(Thrift::BufferedTransport.new(Thrift::Socket.new('www4',
> 9090, 200))
> Nov 11, 2008 6:49:35 PM
> com.facebook.thrift.server.TThreadPoolServer$WorkerProcess run
> SEVERE: Error occurred during processing of message.
> java.lang.NegativeArraySizeException
> at
> com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown
> Source)
> at Cache$search_args.read(Cache.java:344)
> at Cache$Processor$search.process(Cache.java:165)
> at Cache$Processor.process(Cache.java:157)
> at
> com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I can't confirm 100% whether the timeouts in the log are the cause or effect.
> Is it possible that timing out and then re-opening the connection causes
> this somehow?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
