Fantastic, Gabriel. Thanks a lot! Regards, Matthias
On Tuesday, 2013-04-02, Gabriel Reid wrote: > > On 02 Apr 2013, at 21:39, Josh Wills <[email protected]> wrote: > > > In the context of that test (PCollectionGetSizeIT), the EOFException is > > expected-- there isn't any data to read b/c the file is empty. We could > > change up the impl to throw a CrunchRuntimeException instead. > > > > For the cogroup test, that sounds suspiciously like a deep copy issue. > > Gabriel is the most knowledgable person to ask, he always fixes my deep > > copy errors. :) > > Well that's a bizarre coincidence -- I just encountered this object reuse bug > in CoGroup earlier today at work, and was going over mails from today while > running the integration tests after having written a patch for it :-) > > In any case, yes, there is a deep copy issue with CoGroup -- I'm actually > very surprised that this is still in there, as I was sure that I had gone > over CoGroup in the past to make sure that object reuse wasn't an issue. > > In any case, I'll commit the fix shortly. > > - Gabriel > > > > On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <[email protected]> wrote: > > > >> Hi, > >> > >> I'm a bit worried about the IO exceptions we get in our integration > >> test (see [1], grep for EOFException). Since we don't validate the > >> results, there may be data loss, at least in local mode. > >> > >> I have stumbled upon a test case [2] where Crunch seems to reliably > >> lose some data when using Writables, while the same code works fine > >> with Avros. This time, I was able to reproduce this on two machines > >> (with slightly different error messages, ie. [3]). > >> > >> Has anyone seen this before? Is anyone able to reproduce the error > >> or is it just me? :) > >> > >> Thanks, > >> Matthias > >> > >> [1] > >> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull > >> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch > >> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt > >> >
