[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588993#comment-15588993
]
Stefan Egli edited comment on OAK-4581 at 10/20/16 9:49 AM:
------------------------------------------------------------
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork (in [this
branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which
tries to scetch an approach that would use a SwitchingObserer in front of
either a BackgroundObserver/ChangeProcessor pair or a
PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this
approach is that there are quite a few new classes in play. The advantage is
that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.
h4. Prototype based on persistence embedded in the ChangeProcessor
An alternative to the above would be to embedd the persistence directly in the
ChangeProcessor. The contentChanged method there would inspect the queue size
and if too large, not deliver events to the listener anymore, but instead go
via a persistent queue. The advantage is that it's more enclosed in the
ChangeProcessor. The disadvantage is that if a listener would block it could
not prevent the queue from growing (but maybe supporting an indefinitely
blocking listener is not required). I'll look into how such an approach could
be implemented next.
UPDATE: Added a prototype of this second variant too (in [this
branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581-type2]).
h4. Comparison of the approaches
The two approaches basically represent _'head of queue persisting'_
(PersistingEventListener) versus _'tail of queue persisting'_
(SwitchingObserver).
h5. Falling out of the documentMk's diff cache
* 'tail of queue persisting' has the advantage that it supports any type of
storm coming in. It would immediately start persisting newly enqueued commits.
(At the moment those diffs already in the queue are worked off 'slowly' at
onEvent time, but this could be changed to be force-persisted too)
* 'head of queue persisting' is less favorable when falling out of the diff
cache, as it would basically start persisting the head at max speed (ie without
onEvent cost), but perhaps that max speed is slower than new commits coming in.
h5. Storm of commits coming in
* 'tail of queue persisting' again is deterministic for this case as it just
routes incoming commits to persistence for good - no possibility of further
queue growth. And the remaining missing feature of not-yet-force-flushing the
inmemory queue is not a problem here as we remain in the cache.
* 'head of queue persisting' would likely be overwhelmed here: it depends on
how rapidly it can filter/generate/persist off the queue vs how quickly new
commits come in.
h4. Conclusion
While the head-of-queue-persisting/PersistingEventListener seems to be the more
KISS/elegant solution, it has downsides under heavy load. It seems thus better
to do tail-of-queue-persisting with eg the SwitchingObserver.
What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon],
[~tmueller]
was (Author: egli):
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork which tries to scetch an approach that would use a
SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair
or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of
this approach is that there are quite a few new classes in play. The advantage
is that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.
h4. Prototype based on persistence embedded in the ChangeProcessor
An alternative to the above would be to embedd the persistence directly in the
ChangeProcessor. The contentChanged method there would inspect the queue size
and if too large, not deliver events to the listener anymore, but instead go
via a persistent queue. The advantage is that it's more enclosed in the
ChangeProcessor. The disadvantage is that if a listener would block it could
not prevent the queue from growing (but maybe supporting an indefinitely
blocking listener is not required). I'll look into how such an approach could
be implemented next.
> Persistent local journal for more reliable event generation
> -----------------------------------------------------------
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
> Reporter: Chetan Mehrotra
> Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3]
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
