[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger updated OAK-3976:
--
Attachment: OAK-3976-1.patch
Attached a slightly modified patch with fixed typos in tests and minor changes
to the journalPushThreshold field in DocumentNodeStore. I prefer a private
non-static field with a package protected setter for tests. I agree with you on
the empty journal entry. I don't see a reason why the DocumentNodeStore should
create such an empty entry. My patch also includes your suggested changes to
the tests.
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
> Fix For: 1.6, 1.5.16
>
> Attachments: OAK-3976-1.patch, OAK-3976.patch
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-3976:
---
Attachment: OAK-3976.patch
Attaching patch - [^OAK-3976.patch]. Notable difference from earlier version:
* {{JournalEntry}} also tracks if some branch commits have been added - this is
required to avoid the case when a commit pushes the entry and before any other
commit background write tries to push the journal (thus using the same revision
for the entry and logging error that it couldn't push the entry)
* Path counter is incremented if a new {{TreeNode}} is pushed - probably we
should rename it to something like node-count
* Lock around journal push removed
* A bit of refactor of where "numChanges" crosses threshold (now in {{done()}}
method)
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
> Fix For: 1.6, 1.5.16
>
> Attachments: OAK-3976.patch
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-3976:
-
Assignee: Vikas Saurabh (was: Stefan Egli)
Fix Version/s: (was: 1.5.15)
1.5.16
Thanks [~catholicon]! I'm moving this to 1.5.16 then and assigning to you.
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
> Fix For: 1.6, 1.5.16
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Saar updated OAK-3976:
Fix Version/s: 1.6
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
> Fix For: 1.6, 1.5.15
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Saar updated OAK-3976:
Fix Version/s: (was: 1.6)
1.5.15
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
> Fix For: 1.6, 1.5.15
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3976) journal should support large(r) entries
[
https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-3976:
-
Description:
Journal entries are created in the background write. Normally this happens
every second. If for some reason there is a large delay between two background
writes, the number of pending changes can also accumulate. Which can result in
(arbitrary) large single journal entries (ie with large {{_c}} property).
This can cause multiple problems down the road:
* journal gc at this point loads 450 entries - and if some are large this can
result in a very large memory consumption during gc (which can cause severe
stability problems for the VM, if not OOM etc). This should be fixed with
OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
* before OAK-3001 is done (which is currently scheduled after 1.4) what we can
do is reduce the delete batch size (OAK-3975)
* background reads however also read the journal entries and even if
OAK-3001/OAK-3975 are implemented the background read can still cause large
memory consumption. So we need to improve this one way or another.
was:
Journal entries are created in the background write. Normally this happens
every second. If for some reason there is a large delay between two background
writes, the number of journal entries can also accumulate. Which can result in
(arbitrary) large single journal entries (ie with large {{_c}} property).
This can cause multiple problems down the road:
* journal gc at this point loads 450 entries - and if some are large this can
result in a very large memory consumption during gc (which can cause severe
stability problems for the VM, if not OOM etc). This should be fixed with
OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
* before OAK-3001 is done (which is currently scheduled after 1.4) what we can
do is reduce the delete batch size (OAK-3975)
* background reads however also read the journal entries and even if
OAK-3001/OAK-3975 are implemented the background read can still cause large
memory consumption. So we need to improve this one way or another.
> journal should support large(r) entries
> ---
>
> Key: OAK-3976
> URL: https://issues.apache.org/jira/browse/OAK-3976
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
> Fix For: 1.6
>
>
> Journal entries are created in the background write. Normally this happens
> every second. If for some reason there is a large delay between two
> background writes, the number of pending changes can also accumulate. Which
> can result in (arbitrary) large single journal entries (ie with large {{_c}}
> property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can
> result in a very large memory consumption during gc (which can cause severe
> stability problems for the VM, if not OOM etc). This should be fixed with
> OAK-3001 (where we only get the id, thus do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we
> can do is reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if
> OAK-3001/OAK-3975 are implemented the background read can still cause large
> memory consumption. So we need to improve this one way or another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
