> On Dec. 3, 2017, 6:29 p.m., David McLaughlin wrote: > > Do we have any idea how this performs at scale? I'd like to avoid repeating > > the MyBatis work, where we delayed the scale testing until after we had > > committed a bunch of patches.
A fair concern! Until i can perform a more real-world test, here's data using resources readily at hand... Setup: - OS X host running a tool to measure persist/recover - VirtualBox host with 2 CPU, 3 GB RAM running Ubuntu 14.04 and MySQL 5.5.58 Network RTT from host to guest OS tends to be between 250µs and 500µs. The tool calls `persist()` many times, reports stats, then invokes `recover()`. I explored 3 use cases - minimal `SaveQuota` records, large (100 KB) records, and `persist()` calls that batch many records. See results below: ```console -- Simple records (minimal SaveQuota records) Persisted 50000 times in 92653 ms (539 persists/sec) avg 1.85 ms p50 1.27 ms p90 2.54 ms p99 11.56 ms Recovered 50000 records in 354 ms (141,242 records/sec) -- Large records (ScheduledTasks with 100 KB blobs) Persisted 10000 times in 98392 ms (101 persists/sec) avg 9.84 ms p50 6.19 ms p90 20.95 ms p99 36.95 ms Recovered 10000 records in 9605 ms (1,041 records/sec) -- Record batches (each persist is 100 simple records) Persisted 10000 times in 71286 ms (140 persists/sec) avg 7.13 ms p50 4.32 ms p90 14.64 ms p99 32.12 ms Recovered 1000000 records in 25494 ms (39,224 records/sec) ``` This is at best a corse approximation, as server-class hardware should improve these figures; and there would be a tax if MySQL clustering were in play. - Bill ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64288/#review192666 ----------------------------------------------------------- On Dec. 3, 2017, 5:09 p.m., Bill Farner wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64288/ > ----------------------------------------------------------- > > (Updated Dec. 3, 2017, 5:09 p.m.) > > > Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb. > > > Repository: aurora > > > Description > ------- > > Introduces a `Persistence` implementation that uses a SQL database via JDBC. > I've opted to lean towards MySQL SQL dialect, as unfortunately there are > vendor differences for even the very simple SQL used here. > > I chose [HikariCP](https://github.com/brettwooldridge/HikariCP) to serve as a > `DataSource` (connection pool) implementation. We don't really need much out > of a connection pool aside from general connection lifecycle management (i.e. > not for concurrency). I chose this library based on recent development > activity and several positive comparisons to other pools. > > Note that the implementation is not yet wired into the scheduler application. > That will come in a follow-up. > > > Diffs > ----- > > build.gradle af119910e84c48f75f2573ababcfa287c3b986fc > config/spotbugs/excludeFilter.xml 51790cce8d9047e40741f05ee55af15dbdc3065e > > src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorage.java > 85b2113631586f43d854c4d2812f43b7b864d452 > > src/main/java/org/apache/aurora/scheduler/storage/durability/Persistence.java > 9eb862c01bf451252bfbcc7a2eac60d2c965c9f0 > src/main/java/org/apache/aurora/scheduler/storage/log/LogPersistence.java > e70e6051582ca90ae72014626b983bbf4b8d5b48 > src/main/java/org/apache/aurora/scheduler/storage/sql/SqlPersistence.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/sql/SqlPersistenceModule.java > PRE-CREATION > src/main/resources/org/apache/aurora/scheduler/storage/sql/schema.sql > PRE-CREATION > > src/test/java/org/apache/aurora/scheduler/storage/sql/SqlPersistenceTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/64288/diff/1/ > > > Testing > ------- > > > Thanks, > > Bill Farner > >
