[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732050#comment-15732050
]
Vikas Saurabh commented on OAK-3976:
------------------------------------
[~mreutegg], here's roughly what I am thinking of doing:
{code}
diff --git
a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
index 04ecfd7..c168ff0 100644
---
a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
+++
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
@@ -185,6 +185,13 @@ public final class DocumentNodeStore
private boolean disableJournalDiff =
diff --git
a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
index 04ecfd7..c168ff0 100644
---
a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
+++
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
@@ -185,6 +185,13 @@ public final class DocumentNodeStore
private boolean disableJournalDiff =
Boolean.getBoolean(SYS_PROP_DISABLE_JOURNAL);
+ public static final String SYS_PROP_JOURNAL_PUSH_THRESHOLD =
"oak.journalPushThreshold";
+ /**
+ * Threshold for number of paths in journal entry to require a force push
during commit
+ * (instead of at background write)
+ */
+ static int journalPushThreshold = 100000; //non-final to allow for tests.
+
/**
* The document store (might be used by multiple node stores).
*/
@@ -790,6 +797,7 @@ public final class DocumentNodeStore
changes.addChangeSet(getChangeSet(info));
// update head revision
newHead[0] = before.update(c.getRevision());
+ pushJournalEntry(c.getRevision(), false);
setRoot(newHead[0]);
commitQueue.headRevisionChanged();
dispatcher.contentChanged(getRoot(), info);
@@ -2142,19 +2150,32 @@ public final class DocumentNodeStore
return unsavedLastRevisions.persist(this, new
UnsavedModifications.Snapshot() {
@Override
public void acquiring(Revision mostRecent) {
- if (store.create(JOURNAL,
singletonList(changes.asUpdateOp(mostRecent)))) {
- // success: start with a new document
- changes = newJournalEntry();
- } else {
- // fail: log and keep the changes
- LOG.error("Failed to write to journal, accumulating
changes for future write (~" + changes.getMemory()
- + " bytes).");
- }
+ pushJournalEntry(mostRecent, true);
}
}, backgroundOperationLock.writeLock());
}
//-----------------------------< internal
>---------------------------------
+ private ReadWriteLock journalPushLock = new ReentrantReadWriteLock();
+ void pushJournalEntry(Revision r, boolean force) {
+ if (!force && changes.getNumChangedPaths() < journalPushThreshold) {
+ return;
+ }
+ journalPushLock.writeLock().lock();
+ try {
+ if (store.create(JOURNAL, singletonList(changes.asUpdateOp(r)))) {
+ // success: start with a new document
+ changes = newJournalEntry();
+ } else {
+ // fail: log and keep the changes
+ LOG.error("Failed to write to journal, accumulating changes
for future write (~" + changes.getMemory()
+ + " bytes).");
+ }
+ } finally {
+ journalPushLock.writeLock().unlock();
+ }
+
+ }
/**
* Returns the binary size of a property value represented as a JSON or
{code}
A few things that I think were important to take care of:
* Commit (if required) should push journal entry before it sets new head
* minimize journal push critical section
This way things are currently, I'm not completely confident of:
# commit clears journal and pushes an entry with its own revision V/s bkWrite
checked unsavedLastRevs to find the latest rev
#* I'm not sure if commit's rev could be bigger that highest-last-rev. Also,
does it matter?
# BkWrite can now potentially create empty journal entries. Should we avoid
that?
> journal should support large(r) entries
> ---------------------------------------
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
> Affects Versions: 1.3.14
> Reporter: Stefan Egli
> Assignee: Vikas Saurabh
> Fix For: 1.6, 1.5.16
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
