> On Nov. 30, 2017, 6:29 p.m., Bill Farner wrote: > > I have an alternative approach to pose based on learnings from this patch - > > using a DBMS as a key-value document store. This could be ~2 levels of > > abstraction higher and involve implementing `Storage` rather than > > `StreamManager`. Similar to `LogStorage`, we would persist mutations as > > they happen, and only read to sync a replica. Warm standby could be > > implemented by incrementally fetching based on modification timestamps. > > Jordan Ly wrote: > I am +1 to this overall approach. I think this is the natural progression > for durable storage and we get a lot of needs out of the box (namely > transactions). > > Warm standby is definitely nice to have but it isn't as important with > the removal of forced daily failovers for compaction. > > David McLaughlin wrote: > I'm +1 to higher level abstractions. I don't like the idea of having to > assume an underlying Log-based format for durability - and instead have > abstractions that assume (or at least expect) capabilities.
Going directly to a key-value interface sounds also nicer to me than using it behind a log abstraction. One thing I am wondering though: Why do you want to go to a DBMS? While ZK is not a real database, anything in particular that you don't like for this usecase? - Stephan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64126/#review192310 ----------------------------------------------------------- On Nov. 29, 2017, 2:40 a.m., Bill Farner wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64126/ > ----------------------------------------------------------- > > (Updated Nov. 29, 2017, 2:40 a.m.) > > > Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb. > > > Repository: aurora > > > Description > ------- > > This introduces an alternative implementation of `StreamManager`, which is the > primary persistence mechanism used in `LogStorage`. There are two distinct > parts > of this patch: > > - a storage 'business logic' layer that exposes a stream-oriented > append-only > storage interface. This contains components to: > - extract keys for any log `Op` > - hash keys to constrain the keyspace > - [de]serialize `LogEntries` and limit record sizes (using existing > Framing) > - provide simple transactional behavior > - a key-value store interface, with an implementation based on ZooKeeper > > This achieves the following results: > > - no inherent need for periodic Snapshot or failover to cull storage > - atomic multi-operation transactions even for a key-value store that does > not > support multi-key transactions > - storage operations that amortize to a single storage round-trip > (in the case of ZooKeeper, using ZK `multi()`) > - straightforward path to warm standby for failover > > > **Note**: there are still a few rough edges that must be rounded out before > this can be used even in integration tests, but the storage 'business logic' > layers are reasonably polished and well-tested. I opted to use this as a > moment to pause and gather feedback. > > **Reviewers**: I suggest starting with `AbstractStreamManagerTest.java` as it > most succinctly captures the features sought in the new layers. A first pass > review on the general approach would be most useful, after which we can move > on to specifics. > > > Diffs > ----- > > build.gradle af119910e84c48f75f2573ababcfa287c3b986fc > > commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java > a4504b8d1307ca15de3bb92674b8d77c6aea8467 > src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java > 3c12532acca9107f5a54df48fc59348d70af29c6 > src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java > 7ffcf4f471b97e32426c82972472115c5c5c4d02 > src/main/java/org/apache/aurora/scheduler/config/CliOptions.java > b7f43e0d6efbddcac640c3d39c7bc56400e68e68 > > src/main/java/org/apache/aurora/scheduler/storage/log/AbstractStreamTransaction.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/storage/log/Entries.java > ab7cd2d40013827122e27275f557e9c8fff9dbb7 > src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java > 90c5383febc0ac49b0bf8b369fe0107f1fa8a6b3 > src/main/java/org/apache/aurora/scheduler/storage/log/Framing.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java > c8dc7ad141f27318f18afdc440250afc42f0bc6d > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java > ea147c0ba6aaa6d113144be0a8330bd2f73d2453 > > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerFactory.java > 52b2e83efc0c289366b5246aff32553a546b1a70 > > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java > baf2647c54f1f9918139584e5151873a3853e83c > > src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerModule.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/StreamTransaction.java > a51fd18ad00537bf244419442078548d8545a841 > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/AppendOnlyStore.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/HashedKeyValueStore.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStore.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStreamModule.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyedStreamManager.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogEntrySerializingStore.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogOpStore.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/TransactionalKeyValueStore.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyValueStore.java > PRE-CREATION > src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java > 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4 > src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java > 5cb5310ed096ca1fb47b980401e3712948271ac4 > src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java > 3f445595a81a5655c6c486791a9b55d8dc7f2f76 > > src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java > f43a836a4176c465fa3f496c77ef6dc1cbc1691a > > src/test/java/org/apache/aurora/scheduler/storage/log/kv/AbstractStreamManagerTest.java > PRE-CREATION > > src/test/java/org/apache/aurora/scheduler/storage/log/kv/InMemoryStreamManagerTest.java > PRE-CREATION > > src/test/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueNonVolatileStorageTest.java > PRE-CREATION > > src/test/java/org/apache/aurora/scheduler/storage/log/kv/MapKeyValueStore.java > PRE-CREATION > > src/test/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyedStreamManagerTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/64126/diff/1/ > > > Testing > ------- > > > Thanks, > > Bill Farner > >
