> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote: > > Looks good, have you done any manual testing of the change?
I think I found an issue with the source implementation actually, digging into it now. It processes the files but I'm seeing an issue once the directory is empty that I'm having trouble reproducing with unit tests. > On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote: > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, > > line 98 > > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line98> > > > > Why use direct memory here? At least in OpenJDK, when reading from a file if you use a heap buffer it internally allocates a direct buffer then copies to the heap buffer. This is supposed to be a performance optimization. > On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote: > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, > > line 123 > > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line123> > > > > What is the logic here, why 4? 32 bits is the maximum size that a single character should ever take up in Unicode. - Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8596/#review14583 ----------------------------------------------------------- On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8596/ > ----------------------------------------------------------- > > (Updated Dec. 17, 2012, 8:58 a.m.) > > > Review request for Flume. > > > Description > ------- > > Defines EventDeserializer interface and uses it from the spooling source. > Progress is persisted as bytes are read from the underlying file. > > > This addresses bug FLUME-1632. > https://issues.apache.org/jira/browse/FLUME-1632 > > > Diffs > ----- > > flume-ng-core/pom.xml 0224519 > flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java > 37e9ffa > > flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java > 718e1b2 > flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java > 904f22c > > flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java > 8362299 > > flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java > a418935 > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java > 75853a9 > > flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java > afe8ed8 > > flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java > 61824d8 > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java > 806a661 > flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java > 169abe5 > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java > 740bc98 > > flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestSpoolingFileEventParser.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > 6e87b21 > flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro > PRE-CREATION > > flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro > PRE-CREATION > pom.xml b934c1d > > Diff: https://reviews.apache.org/r/8596/diff/ > > > Testing > ------- > > Unit tests pass. > > > Thanks, > > Mike Percy > >
