-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64126/
-----------------------------------------------------------
Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
Repository: aurora
Description
-------
This introduces an alternative implementation of `StreamManager`, which is the
primary persistence mechanism used in `LogStorage`. There are two distinct
parts
of this patch:
- a storage 'business logic' layer that exposes a stream-oriented append-only
storage interface. This contains components to:
- extract keys for any log `Op`
- hash keys to constrain the keyspace
- [de]serialize `LogEntries` and limit record sizes (using existing Framing)
- provide simple transactional behavior
- a key-value store interface, with an implementation based on ZooKeeper
This achieves the following results:
- no inherent need for periodic Snapshot or failover to cull storage
- atomic multi-operation transactions even for a key-value store that does not
support multi-key transactions
- storage operations that amortize to a single storage round-trip
(in the case of ZooKeeper, using ZK `multi()`)
- straightforward path to warm standby for failover
**Note**: there are still a few rough edges that must be rounded out before
this can be used even in integration tests, but the storage 'business logic'
layers are reasonably polished and well-tested. I opted to use this as a
moment to pause and gather feedback.
**Reviewers**: I suggest starting with `AbstractStreamManagerTest.java` as it
most succinctly captures the features sought in the new layers. A first pass
review on the general approach would be most useful, after which we can move on
to specifics.
Diffs
-----
build.gradle af119910e84c48f75f2573ababcfa287c3b986fc
commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java
a4504b8d1307ca15de3bb92674b8d77c6aea8467
src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java
3c12532acca9107f5a54df48fc59348d70af29c6
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
7ffcf4f471b97e32426c82972472115c5c5c4d02
src/main/java/org/apache/aurora/scheduler/config/CliOptions.java
b7f43e0d6efbddcac640c3d39c7bc56400e68e68
src/main/java/org/apache/aurora/scheduler/storage/log/AbstractStreamTransaction.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/Entries.java
ab7cd2d40013827122e27275f557e9c8fff9dbb7
src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java
90c5383febc0ac49b0bf8b369fe0107f1fa8a6b3
src/main/java/org/apache/aurora/scheduler/storage/log/Framing.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java
c8dc7ad141f27318f18afdc440250afc42f0bc6d
src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java
ea147c0ba6aaa6d113144be0a8330bd2f73d2453
src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerFactory.java
52b2e83efc0c289366b5246aff32553a546b1a70
src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java
baf2647c54f1f9918139584e5151873a3853e83c
src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerModule.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/StreamTransaction.java
a51fd18ad00537bf244419442078548d8545a841
src/main/java/org/apache/aurora/scheduler/storage/log/kv/AppendOnlyStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/HashedKeyValueStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStreamModule.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyedStreamManager.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogEntrySerializingStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogOpStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/TransactionalKeyValueStore.java
PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyValueStore.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java
4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4
src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java
5cb5310ed096ca1fb47b980401e3712948271ac4
src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java
3f445595a81a5655c6c486791a9b55d8dc7f2f76
src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java
f43a836a4176c465fa3f496c77ef6dc1cbc1691a
src/test/java/org/apache/aurora/scheduler/storage/log/kv/AbstractStreamManagerTest.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/log/kv/InMemoryStreamManagerTest.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueNonVolatileStorageTest.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/log/kv/MapKeyValueStore.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyedStreamManagerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/64126/diff/1/
Testing
-------
Thanks,
Bill Farner
