On 02 Apr 2013, at 21:39, Josh Wills <[email protected]> wrote: > In the context of that test (PCollectionGetSizeIT), the EOFException is > expected-- there isn't any data to read b/c the file is empty. We could > change up the impl to throw a CrunchRuntimeException instead. > > For the cogroup test, that sounds suspiciously like a deep copy issue. > Gabriel is the most knowledgable person to ask, he always fixes my deep > copy errors. :)
Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup earlier today at work, and was going over mails from today while running the integration tests after having written a patch for it :-) In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised that this is still in there, as I was sure that I had gone over CoGroup in the past to make sure that object reuse wasn't an issue. In any case, I'll commit the fix shortly. - Gabriel > On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <[email protected]> wrote: > >> Hi, >> >> I'm a bit worried about the IO exceptions we get in our integration >> test (see [1], grep for EOFException). Since we don't validate the >> results, there may be data loss, at least in local mode. >> >> I have stumbled upon a test case [2] where Crunch seems to reliably >> lose some data when using Writables, while the same code works fine >> with Avros. This time, I was able to reproduce this on two machines >> (with slightly different error messages, ie. [3]). >> >> Has anyone seen this before? Is anyone able to reproduce the error >> or is it just me? :) >> >> Thanks, >> Matthias >> >> [1] >> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull >> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch >> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt >>
