----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31643/#review76504 -----------------------------------------------------------
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/FileBasedMetadataBackingStore.java <https://reviews.apache.org/r/31643/#comment124057> We want to have MapDB instead of eg. zookeeper because zookeeper is optional, right, or there are other reasons? Maybe also worth to have something like ZookeeperMetadataBackingStore (in a separate task)? flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3ObjectEventReader.java <https://reviews.apache.org/r/31643/#comment124058> Do we know how objects are sorted? Date of creation asc? To make sure, maybe it's worth to sort list of objectSummaries by "getLastModified()" - Pawel On March 2, 2015, 10:26 p.m., Johny Rufus John wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/31643/ > ----------------------------------------------------------- > > (Updated March 2, 2015, 10:26 p.m.) > > > Review request for Flume. > > > Bugs: FLUME-2437 > https://issues.apache.org/jira/browse/FLUME-2437 > > > Repository: flume-git > > > Description > ------- > > S3 Source, initial version > > > Diffs > ----- > > > flume-ng-core/src/main/java/org/apache/flume/serialization/FileStreamCreator.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableGenericInputStream.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/StreamCreator.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java > d1240fb > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableGenericInputStream.java > PRE-CREATION > flume-ng-dist/pom.xml a083fe2 > flume-ng-sources/flume-s3-source/pom.xml PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/FileBasedMetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/InMemoryMetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStoreFactory.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3ObjectEventReader.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3Source.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3SourceConfigurationConstants.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3StreamCreator.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/test/java/org/apache/flume/source/s3/TestS3Source.java > PRE-CREATION > flume-ng-sources/pom.xml ab8eca4 > pom.xml ea7ffe3 > > Diff: https://reviews.apache.org/r/31643/diff/ > > > Testing > ------- > > TestResettableGenericInputStream and > TestS3Source > > Manual testing of scenarios: > 1. Created multiple files in S3 Bucket - make sure the source processes all > the files > 2. Add more files, after the S3 source starts - make sure the newly created > S3 objects are processed > 3. Stop the source after a few files are processed - make sure on restart, > the source only processes the rest of the unprocessed files > 4. Stop the source in the middle of processing a file - make sure the postion > tracker is read on re-start and processing of the partial file continues from > where it was last marked > > > Thanks, > > Johny Rufus John > >
