Hemant Thanks for the quick response - I'll admit I hadn't resynced for a week or so! Agreed that non-blocking reads make more sense for the client.
We're still testing with a patched version of packet-0.1.5 and the svn release of backgroundrb because of the problems we hit getting RoR model objects to load properly on the worker tasks with the latest code - have you found a fix for this yet? Mike -----Original Message----- From: Hemant Kumar [mailto:[EMAIL PROTECTED] Sent: 10 June 2008 16:01 To: Mike Evans Cc: [email protected] Subject: Re: Backgroundrb fixes for transfering large amounts of data Mike Evans wrote: > > Hemant > > We've continued testing our application with backgroundrb and found a > couple of other problems when transfering large amounts of data. Both > of these problems are still present in the github version of code. > > The first problem is an exception in the Marshal.load call in the > receive_data method in the Packet::MetaPimp class. The root cause is > in the BinParser module, in the arm of code handling the parser state > 1 (reading in the data). The issue is that, at the marked line of > code the pack_data array will be at most @numeric_length entries > because of the format string passed to the unpack call. This results > in the code dropping a chunk of data and then hitting the exception in > a subsequent Marshal.load call. > > elsif @parser_state == 1 > pack_data,remaining = new_data.unpack("[EMAIL PROTECTED]") > if pack_data.length < @numeric_length > @data << pack_data > @numeric_length = @numeric_length - pack_data.length > elsif pack_data.length == @numeric_length <======== this > should be "elsif remaining.length == 0" > @data << pack_data > extracter_block.call(@data.join) > @data = [] > @parser_state = 0 > @length_string = "" > @numeric_length = 0 > else > @data << pack_data > extracter_block.call(@data.join) > @data = [] > @parser_state = 0 > @length_string = "" > @numeric_length = 0 > extract(remaining,&extracter_block) > end > end > > The second problem we hit was ask_status repeatedly returning nil. > The root cause of this problem is in the read_object method of the > BackgrounDRb::WorkerProxy class when a data record is large enough to > cause connection.read_nonblock to throw the Errno::EAGAIN exception > multiple times. We changed the code to make sure read_nonblock is > called repeatedly until the tokenizer finds a complete record, and > this fixed the problem. > > def read_object > begin > while (true) > sock_data = "" > begin > while(sock_data << @connection.read_nonblock(1023)); end > rescue Errno::EAGAIN > @tokenizer.extract(sock_data) { |b_data| return b_data } > end > end > rescue > raise BackgrounDRb::BdrbConnError.new("Not able to connect") > end > end > > Regards, Mike > If you update to THE latest github version of BackgrounDRb, you will find that above thing is already fixed and read is no more nonblocking for clients ( blocking read makes much more sense for clients ). Also, I have made BinParser class to be iterative ( just for better stability, assuming your data is large enough to throw StackLevel Too deep errors), but thats not yet pushed to master version of packet. I will implement your fix tonight, when I push the changes to master repository of packet. _______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
