[
https://issues.apache.org/jira/browse/AURORA-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220221#comment-14220221
]
Maxim Khutornenko commented on AURORA-942:
------------------------------------------
Still, using something that was not designed to be a database [1] feels like a
forced approach even considering our moderate (currently) data footprint:
{quote}
ZooKeeper was not designed to be a general database or large object store.
Instead, it manages coordination data. This data can come in the form of
configuration, status information, rendezvous, etc. A common property of the
various forms of coordination data is that they are relatively small: measured
in kilobytes. The ZooKeeper client and the server implementations have sanity
checks to ensure that znodes have less than 1M of data, but the data should be
much less than that on average. Operating on relatively large data sizes will
cause some operations to take much more time than others and will affect the
latencies of some operations because of the extra time needed to move more data
over the network and onto storage media.
{quote}
The fact that we have chunking is great but managing sequencing would be
completely on us, perhaps following the sequenced node approach [2]. Not saying
it's impossible, just more/different things to worry about, including the ZK
data backup.
[1] -
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Data+Access:
[2] - http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_Queues
> Explore using a replicated log on top of ZooKeeper
> --------------------------------------------------
>
> Key: AURORA-942
> URL: https://issues.apache.org/jira/browse/AURORA-942
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
> Reporter: Bill Farner
> Priority: Minor
>
> The scheduler uses the replicated log implementation provided by mesos
> (native libmesos.so). It would be interesting to compare this against a
> replacement that sllows us to:
> - shed code to implement backups and recovery
> - remove one use of a dynamically-linked native library
> - use a store that allows non-leaders to read, for faster recovery and
> serving from non-active members
> - avoid the need for periodic failover (we currently have to do this to
> induce compaction in LevelDB and minimize log replay time)
> At first glance, it seems like it would be relatively straightforward to come
> up with a Log implementation \[1\] that persists transactions as nodes in
> ZooKeeper. This would enable all the above results.
> \[1\]
> https://github.com/apache/incubator-aurora/blob/10da38a3a0ad6ebbee055c26adc3ed3437ec3930/src/main/java/org/apache/aurora/scheduler/log/Log.java#L26
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
