----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/#review56762 -----------------------------------------------------------
Thanks for adding the documentation! I'm a +1 once the copying code is less performant but more readable/concise. src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java <https://reviews.apache.org/r/26478/#comment97167> Please use a better variable name. docs/scheduler-storage.md <https://reviews.apache.org/r/26478/#comment97161> > Most users will want to enable both compression and deduplication. I suggest you yank this sentence out of this section, and add to the opening paragraph: > The scheduler has two optimizations to reduce the size of snapshots and thus improve snapshot performance: compression and deduplication. Most users will want to enable both compression and deduplication. src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java <https://reviews.apache.org/r/26478/#comment97165> 'reduplicate' doesn't sit well with me. Perhaps 'normalize' and 'denormalize' are more standard terms that apply? I don't feel too strongly, so don't change it if they seem equally good to you. src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java <https://reviews.apache.org/r/26478/#comment97172> This line is not covered in tests. Please address. However, i suggest you implement this as below, and inline. ScheduledTask partialScheduledTask = scheduledTask.deepCopy(); partialScheduledTask.getAssignedTask().unsetTaskConfig(); return partialScheduledTask; - Bill Farner On Oct. 15, 2014, 1:32 a.m., Kevin Sweeney wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26478/ > ----------------------------------------------------------- > > (Updated Oct. 15, 2014, 1:32 a.m.) > > > Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji. > > > Bugs: AURORA-722 > https://issues.apache.org/jira/browse/AURORA-722 > > > Repository: aurora > > > Description > ------- > > Add a new format for deduplicated storage snapshots. Microbenchmarks show a > 10x deduplication ratio on Twitter's production snapshots. > > This format is backwards-incompatible, so this patch introduces a flag to > control its use (defaulting off). > > This only changes the format used to write to the replicated log (where time > is of the essence since all writes are done holding the global storage lock) > - the format of backups written to disk is unchanged, as backups don't hold > the lock. > > > Diffs > ----- > > config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 > docs/scheduler-storage.md PRE-CREATION > src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java > 65e986eaa2c4193431ca048425a1ed3ab60f5882 > src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java > 7239a6a5eb5479e395e16423c83fdf80a77e5a83 > src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java > 4b50e2069407dc263b4fc93f1827d3a8836253bf > src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java > f806297d1d0700155c976743f936b2b8a3a390fb > src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java > 769348e6b8a5c701734afff391b1c77de35222c6 > > src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java > 22db80eaf34fe736fa5a3a9289836c9ac9e59906 > > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java > e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a > src/main/thrift/org/apache/aurora/gen/storage.thrift > 5350ec945fbe028ee4641683815a068ce00b5efc > src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java > 39729b374fe4e383f9b5ada7d016923766df9af7 > src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java > 7a8c3b882633376a1bf6a78616d55aaa7401d13f > > src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/26478/diff/ > > > Testing > ------- > > ./gradlew -Pq build > > > Thanks, > > Kevin Sweeney > >