[jira] [Commented] (OAK-7262) LockBasedScheduler#getHeadNodeState poor performance due to lock contention in commitTimeHistogram implementation
[
https://issues.apache.org/jira/browse/OAK-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364389#comment-16364389
]
Valentin Olteanu commented on OAK-7262:
---
FWIU the patch is not related to compaction, that's why I was thinking of
having a dedicated stress test in the benchmark that reproduces this particular
case...
> LockBasedScheduler#getHeadNodeState poor performance due to lock contention
> in commitTimeHistogram implementation
> -
>
> Key: OAK-7262
> URL: https://issues.apache.org/jira/browse/OAK-7262
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> With OAK-4732, we slightly prioritised reads, allowing callers of
> {{LockBasedScheduler#getHeadNodeState}} to wait for a certain amount of time
> on the commit semaphore, in order to fetch the latest head. The amount of
> time to wait for was computed as the median of the latest 1000 commits, but
> the implementation was flexible enough to replace the 0.5 quantile with a
> different value.
> The initial implementation used {{SynchronizedDescriptiveStatistics}} from
> {{commons-math3}}. In OAK-6430, we replaced that with a {{Histogram}} backed
> by a {{SlindingWindowReservoir}} from the dropwizard metric library
> ({{io.dropwizard.metrics}}).
> One of the problems observed with the current implementation is that, under
> heavy loading, the performance of {{LockBasedScheduler#getHeadNodeState}}
> decreases due to lock contention in {{commitTimeHistogram}} implementation.
> Namely, {{SlidingWindowReservoir#getSnapshot}} seems to be a bottleneck. It
> copies an array of 1000 elements at each invocation of
> {{LockBasedScheduler#getHeadNodeState}}, acquiring and releasing a lock for
> each element in the array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7262) LockBasedScheduler#getHeadNodeState poor performance due to lock contention in commitTimeHistogram implementation
[
https://issues.apache.org/jira/browse/OAK-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363844#comment-16363844
]
Valentin Olteanu commented on OAK-7262:
---
[~dulceanu], is there a stress test you could image that shows the contention
before the patch?
> LockBasedScheduler#getHeadNodeState poor performance due to lock contention
> in commitTimeHistogram implementation
> -
>
> Key: OAK-7262
> URL: https://issues.apache.org/jira/browse/OAK-7262
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> With OAK-4732, we slightly prioritised reads, allowing callers of
> {{LockBasedScheduler#getHeadNodeState}} to wait for a certain amount of time
> on the commit semaphore, in order to fetch the latest head. The amount of
> time to wait for was computed as the median of the latest 1000 commits, but
> the implementation was flexible enough to replace the 0.5 quantile with a
> different value.
> The initial implementation used {{SynchronizedDescriptiveStatistics}} from
> {{commons-math3}}. In OAK-6430, we replaced that with a {{Histogram}} backed
> by a {{SlindingWindowReservoir}} from the dropwizard metric library
> ({{io.dropwizard.metrics}}).
> One of the problems observed with the current implementation is that, under
> heavy loading, the performance of {{LockBasedScheduler#getHeadNodeState}}
> decreases due to lock contention in {{commitTimeHistogram}} implementation.
> Namely, {{SlidingWindowReservoir#getSnapshot}} seems to be a bottleneck. It
> copies an array of 1000 elements at each invocation of
> {{LockBasedScheduler#getHeadNodeState}}, acquiring and releasing a lock for
> each element in the array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7224) oak-run check should have an option to check the segments checksums
[
https://issues.apache.org/jira/browse/OAK-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346566#comment-16346566
]
Valentin Olteanu commented on OAK-7224:
---
If this is a completely different type of check, e.g. only checksum of
segments, I would create another command: {{check-segments}}. But if it's doing
both the classical check and the segments checksum, then it's ok to have a
parameter...
> oak-run check should have an option to check the segments checksums
> ---
>
> Key: OAK-7224
> URL: https://issues.apache.org/jira/browse/OAK-7224
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run, segment-tar
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Minor
> Labels: tooling
> Fix For: 1.9.0, 1.10
>
>
> {{oak-run check}} does currently *not* check the checksums of the segments.
> As a consequence, there is no quick way of determining the state of the
> repository (corrupt/valid), after corrupting some random node record, as we
> currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that,
> there needs to be an attempt to read the corrupt record as part of a
> traversal.
> An easier way would be to have a new dedicated option for this (i.e.,
> {{--segments}}) which checks by default the content of segments against the
> checksums from all the tar files in the specified location. Additionally, it
> could accept as an argument a list of tar files, the segments of which to be
> checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-3855) oak-run compact should check segment version
[
https://issues.apache.org/jira/browse/OAK-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294770#comment-16294770
]
Valentin Olteanu commented on OAK-3855:
---
[~mduerig], is the new parameter already documented somewhere?
> oak-run compact should check segment version
>
>
> Key: OAK-3855
> URL: https://issues.apache.org/jira/browse/OAK-3855
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: run
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: gc, resilience, tooling
> Fix For: 1.3.15, 1.4
>
>
> Off line compaction should exit with a warning if run on a store with non
> matching segment version. It should provide a {{--force}} option to override
> this behaviour such that it can still be used for explicit upgrading.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-6836) OnRC report
[
https://issues.apache.org/jira/browse/OAK-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290805#comment-16290805
]
Valentin Olteanu edited comment on OAK-6836 at 12/14/17 1:29 PM:
-
I've started a patch in
https://github.com/volteanu/jackrabbit-oak/tree/OAK-6836 with the [first
commit|https://github.com/volteanu/jackrabbit-oak/commit/c4e68d4a7c5a8df27738a418589c71cc5199302e]
that adds the field with just a few info:
!Screen Shot 2017-12-14 at 1.55.40 PM.png|thumbnail!
In the logs it looks like this:
{code}
14.12.2017 13:39:50.518 *INFO* [TarMK revision gc
[/Users/volteanu/workspace/test/author/crx-quickstart/repository/segmentstore]]
org.apache.jackrabbit.oak.segment.file.FileStore GCReport{gcCount=1,
gcType=null, result=null}
{code}
[~mduerig], [~frm], [~dulceanu], WDYT? I would like to know if the start looks
good before adding all the data.
One thing I'm not sure about is the format of the string. It should be human
readable, but also parsable/grep-able. Maybe json?
was (Author: volteanu):
I've started a patch in
https://github.com/volteanu/jackrabbit-oak/tree/OAK-6836 with the [first
commit|https://github.com/volteanu/jackrabbit-oak/commit/c4e68d4a7c5a8df27738a418589c71cc5199302e]
that adds the field with just a few info:
!Screen Shot 2017-12-14 at 1.55.40 PM.png|thumbnail!
In the logs it looks like this:
{code}
14.12.2017 13:39:50.518 *INFO* [TarMK revision gc
[/Users/volteanu/workspace/test/author/crx-quickstart/repository/segmentstore]]
org.apache.jackrabbit.oak.segment.file.FileStore GCReport{gcCount=1,
gcType=null, result=null}
{code}
[~mduerig], [~frm], [~dulceanu], WDYT? I would like to know if the start looks
good before adding all the data.
One thing I'm not sure about is the format of the string. It should be human
readable, but also parsable/grep-able. Maybe json?
> OnRC report
> ---
>
> Key: OAK-6836
> URL: https://issues.apache.org/jira/browse/OAK-6836
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Priority: Minor
> Labels: production
> Attachments: Screen Shot 2017-12-14 at 1.55.40 PM.png
>
>
> Currently, the information regarding an online revision cleanup execution is
> scattered across multiple log lines and partially available in the attributes
> of {{SegmentRevisionGarbageCollection}} MBean.
> While useful for debugging, this is hard to grasp for users that need to
> understand the full process to be able to read it.
> The idea would be to create a "report" with all the details of an execution
> and output it at the end - write to log, but also store it in the MBean, from
> where it can be consumed by monitoring and health checks.
> In the MBean, this would replace the _Last*_ attributes.
> In the logs, this could replace all the intermediary logs (switch them to
> DEBUG).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6941) Compatibility matrix for oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251212#comment-16251212
]
Valentin Olteanu commented on OAK-6941:
---
Probably the simplest solution is through documentation: a page containing the
sparse matrix of known compatible versions.
Another format would be a table with the most recent known compatible version.
> Compatibility matrix for oak-run compact
>
>
> Key: OAK-6941
> URL: https://issues.apache.org/jira/browse/OAK-6941
> Project: Jackrabbit Oak
> Issue Type: Documentation
> Components: run, segment-tar
>Reporter: Valentin Olteanu
>
> h4. Problem statement
> For compacting the segmentstore using {{oak-run}}, the safest option is to
> use the same version of {{oak-run}} as the Oak version used to generate the
> repository. Yet, sometimes, a newer {{oak-run}} version is recommended to
> benefit of bug fixes and improvements, but not every combination of source
> repo and oak-run is safe to use and the user needs a way to check the
> compatibility. Thus, the users need a tool that guides the decision of which
> version to use.
> h4. Requirements
> * Easy to decide what {{oak-run}} version should be used for a certain Oak
> version
> * Up to date with the latest releases
> * Machine readable for scripting
> * Include details on the benefits of using a certain version (release notes)
> * Blacklist of versions that should not be used (with alternatives)
> h4. Solution
> TBD
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6941) Compatibility matrix for oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-6941:
--
Description:
h4. Problem statement
For compacting the segmentstore using {{oak-run}}, the safest option is to use
the same version of {{oak-run}} as the Oak version used to generate the
repository. Yet, sometimes, a newer {{oak-run}} version is recommended to
benefit of bug fixes and improvements, but not every combination of source repo
and oak-run is safe to use and the user needs a way to check the compatibility.
Thus, the users need a tool that guides the decision of which version to use.
h4. Requirements
* Easy to decide what {{oak-run}} version should be used for a certain Oak
version
* Up to date with the latest releases
* Machine readable for scripting
* Include details on the benefits of using a certain version (release notes)
* Blacklist of versions that should not be used (with alternatives)
h4. Solution
TBD
> Compatibility matrix for oak-run compact
>
>
> Key: OAK-6941
> URL: https://issues.apache.org/jira/browse/OAK-6941
> Project: Jackrabbit Oak
> Issue Type: Documentation
> Components: run, segment-tar
>Reporter: Valentin Olteanu
>
> h4. Problem statement
> For compacting the segmentstore using {{oak-run}}, the safest option is to
> use the same version of {{oak-run}} as the Oak version used to generate the
> repository. Yet, sometimes, a newer {{oak-run}} version is recommended to
> benefit of bug fixes and improvements, but not every combination of source
> repo and oak-run is safe to use and the user needs a way to check the
> compatibility. Thus, the users need a tool that guides the decision of which
> version to use.
> h4. Requirements
> * Easy to decide what {{oak-run}} version should be used for a certain Oak
> version
> * Up to date with the latest releases
> * Machine readable for scripting
> * Include details on the benefits of using a certain version (release notes)
> * Blacklist of versions that should not be used (with alternatives)
> h4. Solution
> TBD
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6941) Compatibility matrix for oak-run compact
Valentin Olteanu created OAK-6941: - Summary: Compatibility matrix for oak-run compact Key: OAK-6941 URL: https://issues.apache.org/jira/browse/OAK-6941 Project: Jackrabbit Oak Issue Type: Documentation Components: run, segment-tar Reporter: Valentin Olteanu -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6836) OnRC report
[
https://issues.apache.org/jira/browse/OAK-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205973#comment-16205973
]
Valentin Olteanu commented on OAK-6836:
---
[~mduerig], [~frm], [~dulceanu], what do you think about the general idea?
I have some WIP code (independent to segment-tar) that I use to parse the logs
and output such reports (I needed it to analyze several days of logs). I could
try to adapt the POC to the segment tar and come up with a patch, but I want to
know if you think it's worth going this way.
> OnRC report
> ---
>
> Key: OAK-6836
> URL: https://issues.apache.org/jira/browse/OAK-6836
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
> Fix For: 1.8
>
>
> Currently, the information regarding an online revision cleanup execution is
> scattered across multiple log lines and partially available in the attributes
> of {{SegmentRevisionGarbageCollection}} MBean.
> While useful for debugging, this is hard to grasp for users that need to
> understand the full process to be able to read it.
> The idea would be to create a "report" with all the details of an execution
> and output it at the end - write to log, but also store it in the MBean, from
> where it can be consumed by monitoring and health checks.
> In the MBean, this would replace the _Last*_ attributes.
> In the logs, this could replace all the intermediary logs (switch them to
> DEBUG).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6836) OnRC report
Valentin Olteanu created OAK-6836:
-
Summary: OnRC report
Key: OAK-6836
URL: https://issues.apache.org/jira/browse/OAK-6836
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segment-tar
Reporter: Valentin Olteanu
Fix For: 1.8
Currently, the information regarding an online revision cleanup execution is
scattered across multiple log lines and partially available in the attributes
of {{SegmentRevisionGarbageCollection}} MBean.
While useful for debugging, this is hard to grasp for users that need to
understand the full process to be able to read it.
The idea would be to create a "report" with all the details of an execution and
output it at the end - write to log, but also store it in the MBean, from where
it can be consumed by monitoring and health checks.
In the MBean, this would replace the _Last*_ attributes.
In the logs, this could replace all the intermediary logs (switch them to
DEBUG).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6784) Exceptions are inhibited in oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-6784:
--
Affects Version/s: 1.6.5
> Exceptions are inhibited in oak-run compact
> ---
>
> Key: OAK-6784
> URL: https://issues.apache.org/jira/browse/OAK-6784
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: run, segment-tar
>Affects Versions: 1.7.8, 1.6.5
>Reporter: Valentin Olteanu
> Fix For: 1.8
>
>
> Exceptions thrown by {{oak-run compact}} are inhibited so the exit code of
> the command is not correct in case of error.
> Example:
> {code}
> $ java -jar oak-run-1.7.8-R1809845.jar compact test-oak-run/
> Apache Jackrabbit Oak 1.7.8-R1809845
> Compacting test-oak-run-6.3.0
> With default access mode
> before
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> -> compacting
> org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException:
> Using a too recent version of oak-segment-tar
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:165)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:135)
> at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
> at
> org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
> at
> org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
> after
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> removed files []
> added files []
> Compaction succeeded in 211.9 ms (0s).
> {code}
> A quick fix would be to wrap the exception into a {{RuntimeException}}:
> {code}
> ---
> a/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> +++
> b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> @@ -127,7 +127,7 @@ public class Compact implements Runnable {
> try {
> compact();
> } catch (Exception e) {
> -e.printStackTrace();
> +throw new RuntimeException("Failed to run compact", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6784) Exceptions are inhibited in oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192997#comment-16192997
]
Valentin Olteanu commented on OAK-6784:
---
The same happens in {{oak-run-1.6.5}} so it might be worth backporting.
> Exceptions are inhibited in oak-run compact
> ---
>
> Key: OAK-6784
> URL: https://issues.apache.org/jira/browse/OAK-6784
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: run, segment-tar
>Affects Versions: 1.7.8, 1.6.5
>Reporter: Valentin Olteanu
> Fix For: 1.8
>
>
> Exceptions thrown by {{oak-run compact}} are inhibited so the exit code of
> the command is not correct in case of error.
> Example:
> {code}
> $ java -jar oak-run-1.7.8-R1809845.jar compact test-oak-run/
> Apache Jackrabbit Oak 1.7.8-R1809845
> Compacting test-oak-run-6.3.0
> With default access mode
> before
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> -> compacting
> org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException:
> Using a too recent version of oak-segment-tar
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:165)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:135)
> at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
> at
> org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
> at
> org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
> after
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> removed files []
> added files []
> Compaction succeeded in 211.9 ms (0s).
> {code}
> A quick fix would be to wrap the exception into a {{RuntimeException}}:
> {code}
> ---
> a/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> +++
> b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> @@ -127,7 +127,7 @@ public class Compact implements Runnable {
> try {
> compact();
> } catch (Exception e) {
> -e.printStackTrace();
> +throw new RuntimeException("Failed to run compact", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6784) Exceptions are inhibited in oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192935#comment-16192935
]
Valentin Olteanu commented on OAK-6784:
---
Output with the proposed fix:
{code}
$ java -jar oak-run-1.8-SNAPSHOT.jar compact exception-test/
Apache Jackrabbit Oak 1.8-SNAPSHOT
Compacting exception-test
With default access mode
before
Thu Oct 05 15:43:40 CEST 2017, journal.log
Thu Oct 05 15:43:41 CEST 2017, data0a.tar
Thu Oct 05 15:43:41 CEST 2017, manifest
Thu Oct 05 15:43:41 CEST 2017, repo.lock
size 119.1 MB (119133142 bytes)
-> compacting
Compaction failure stack trace:
java.lang.RuntimeException: Failed to run compact
at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:131)
at
org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
at
org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
Caused by:
org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException: Using
a too recent version of oak-segment-tar
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
at
org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
at
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
at
org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:166)
at
org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:136)
at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
... 3 more
Compaction failed in 139.8 ms (0s).
{code}
> Exceptions are inhibited in oak-run compact
> ---
>
> Key: OAK-6784
> URL: https://issues.apache.org/jira/browse/OAK-6784
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: run, segment-tar
>Affects Versions: 1.7.8
>Reporter: Valentin Olteanu
> Fix For: 1.8
>
>
> Exceptions thrown by {{oak-run compact}} are inhibited so the exit code of
> the command is not correct in case of error.
> Example:
> {code}
> $ java -jar oak-run-1.7.8-R1809845.jar compact test-oak-run/
> Apache Jackrabbit Oak 1.7.8-R1809845
> Compacting test-oak-run-6.3.0
> With default access mode
> before
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> -> compacting
> org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException:
> Using a too recent version of oak-segment-tar
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:165)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:135)
> at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
> at
> org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
> at
> org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
> after
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> removed files []
> added files []
> Compaction succeeded in 211.9 ms (0s).
> {code}
> A quick fix would be to wrap the exception into a {{RuntimeException}}:
> {code}
> ---
> a/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> +++
> b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> @@ -127,7 +127,7 @@ public class Compact implements Runnable {
> try {
> compact();
> } catch (Exception e) {
> -
[jira] [Commented] (OAK-6784) Exceptions are inhibited in oak-run compact
[
https://issues.apache.org/jira/browse/OAK-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192930#comment-16192930
]
Valentin Olteanu commented on OAK-6784:
---
ping [~mduerig], [~frm], [~dulceanu].
> Exceptions are inhibited in oak-run compact
> ---
>
> Key: OAK-6784
> URL: https://issues.apache.org/jira/browse/OAK-6784
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: run, segment-tar
>Affects Versions: 1.7.8
>Reporter: Valentin Olteanu
> Fix For: 1.8
>
>
> Exceptions thrown by {{oak-run compact}} are inhibited so the exit code of
> the command is not correct in case of error.
> Example:
> {code}
> $ java -jar oak-run-1.7.8-R1809845.jar compact test-oak-run/
> Apache Jackrabbit Oak 1.7.8-R1809845
> Compacting test-oak-run-6.3.0
> With default access mode
> before
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> -> compacting
> org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException:
> Using a too recent version of oak-segment-tar
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
> at
> org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:165)
> at
> org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:135)
> at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
> at
> org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
> at
> org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
> after
> Thu Oct 05 15:14:22 CEST 2017, journal.log
> Thu Oct 05 15:14:23 CEST 2017, data0a.tar
> Thu Oct 05 15:14:23 CEST 2017, manifest
> Thu Oct 05 15:14:23 CEST 2017, repo.lock
> size 119.1 MB (119133142 bytes)
> removed files []
> added files []
> Compaction succeeded in 211.9 ms (0s).
> {code}
> A quick fix would be to wrap the exception into a {{RuntimeException}}:
> {code}
> ---
> a/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> +++
> b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
> @@ -127,7 +127,7 @@ public class Compact implements Runnable {
> try {
> compact();
> } catch (Exception e) {
> -e.printStackTrace();
> +throw new RuntimeException("Failed to run compact", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6784) Exceptions are inhibited in oak-run compact
Valentin Olteanu created OAK-6784:
-
Summary: Exceptions are inhibited in oak-run compact
Key: OAK-6784
URL: https://issues.apache.org/jira/browse/OAK-6784
Project: Jackrabbit Oak
Issue Type: Bug
Components: run, segment-tar
Affects Versions: 1.7.8
Reporter: Valentin Olteanu
Fix For: 1.8
Exceptions thrown by {{oak-run compact}} are inhibited so the exit code of the
command is not correct in case of error.
Example:
{code}
$ java -jar oak-run-1.7.8-R1809845.jar compact test-oak-run/
Apache Jackrabbit Oak 1.7.8-R1809845
Compacting test-oak-run-6.3.0
With default access mode
before
Thu Oct 05 15:14:22 CEST 2017, journal.log
Thu Oct 05 15:14:23 CEST 2017, data0a.tar
Thu Oct 05 15:14:23 CEST 2017, manifest
Thu Oct 05 15:14:23 CEST 2017, repo.lock
size 119.1 MB (119133142 bytes)
-> compacting
org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException: Using
a too recent version of oak-segment-tar
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkStoreVersion(ManifestChecker.java:81)
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkManifest(ManifestChecker.java:70)
at
org.apache.jackrabbit.oak.segment.file.ManifestChecker.checkAndUpdateManifest(ManifestChecker.java:51)
at
org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:191)
at
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:343)
at
org.apache.jackrabbit.oak.segment.tool.Compact.newFileStore(Compact.java:165)
at
org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:135)
at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:128)
at
org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:183)
at
org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:93)
at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
after
Thu Oct 05 15:14:22 CEST 2017, journal.log
Thu Oct 05 15:14:23 CEST 2017, data0a.tar
Thu Oct 05 15:14:23 CEST 2017, manifest
Thu Oct 05 15:14:23 CEST 2017, repo.lock
size 119.1 MB (119133142 bytes)
removed files []
added files []
Compaction succeeded in 211.9 ms (0s).
{code}
A quick fix would be to wrap the exception into a {{RuntimeException}}:
{code}
---
a/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
+++
b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
@@ -127,7 +127,7 @@ public class Compact implements Runnable {
try {
compact();
} catch (Exception e) {
-e.printStackTrace();
+throw new RuntimeException("Failed to run compact", e);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6560) Sidegrade uses too much memory
[ https://issues.apache.org/jira/browse/OAK-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138613#comment-16138613 ] Valentin Olteanu commented on OAK-6560: --- [~tomek.rekawek], in this case, could you also implement some sort of progress logging (maybe at each write), similar to the "nodes copied"? I always felt that the commit phase was too silent and never knew if it the migration was blocked or not. > Sidegrade uses too much memory > -- > > Key: OAK-6560 > URL: https://issues.apache.org/jira/browse/OAK-6560 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.8, 1.7.7 > > Attachments: OAK-6560.patch, OAK-6560.patch > > > Repo migration from old segment to oak segment tar requires heap memory > proportional to the repository size, which does not scale well for > large-scale repositories. Also, some structures (e.g. lots of direct > children) have a big impact on the memory footprint. > For some repositories, the sidegrade requires as much as 40GB heap to perform > the migration, which might not be an option for most of the customers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6452) IllegalStateException: too much data for a segment during oak-upgrade from segment to segment-tar
[
https://issues.apache.org/jira/browse/OAK-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu resolved OAK-6452.
---
Resolution: Cannot Reproduce
I've tried again the same command, but was not able to reproduce - it failed
directly with OOM. Thus, I'm closing this issue and will open another one if
needed.
> IllegalStateException: too much data for a segment during oak-upgrade from
> segment to segment-tar
> -
>
> Key: OAK-6452
> URL: https://issues.apache.org/jira/browse/OAK-6452
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, upgrade
>Affects Versions: 1.7.3
>Reporter: Valentin Olteanu
>Assignee: Valentin Olteanu
>Priority: Critical
> Fix For: 1.7.6
>
>
> During the migration of a big repo from the {{old-segment}} format to
> {{segment-tar}} using {{oak-upgrade-1.7.3}}, I've got the following error:
> {code}
> 14.07.2017 09:05:51.920 [main] *INFO*
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade - Copying node
> #89333: /oak:index/uuid/:index/a9f9a3ed-6183-4e9e-9480-1b4fd196a829
> 14.07.2017 10:00:27.957 [TarMK flush
> [extracted/crx-quickstart/repository-oak-upgrade/segmentstore]] *ERROR*
> org.apache.jackrabbit.oak.segment.file.SafeRunnable - Uncaught exception in
> TarMK flush [extracted/crx-quickstart/repository-oak-upgrade/segmentstore]
> java.lang.IllegalStateException: too much data for a segment
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:322)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush(SegmentBufferWriterPool.java:142)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.DefaultSegmentWriter.flush(DefaultSegmentWriter.java:138)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:345)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:342)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(TarRevisions.java:213)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(TarRevisions.java:201)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.flush(FileStore.java:342)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:242)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_131]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_131]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_131]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> 14.07.2017 10:00:28.448 [main] *INFO*
> org.apache.jackrabbit.oak.segment.file.FileStore - TarMK closed:
> extracted/crx-quickstart/repository-oak-upgrade/segmentstore
> 14.07.2017 10:00:28.658 [main] *INFO*
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore - TarMK closed:
> extracted/crx-quickstart/repository/segmentstore
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at
> org.apache.jackrabbit.oak.segment.RecordType.values(RecordType.java:24)
> at
> org.apache.jackrabbit.oak.segment.ImmutableRecordNumbers$1$1.getType(ImmutableRecordNumbers.java:86)
> at
> org.apache.jackrabbit.oak.segment.Segment.forEachRecord(Segment.java:703)
> at
> org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readBinaryReferences(AbstractFileStore.java:277)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(FileStore.java:511)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:356)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.prepare(SegmentBufferWriter.java:423)
> at
>
[jira] [Commented] (OAK-6452) IllegalStateException: too much data for a segment during oak-upgrade from segment to segment-tar
[
https://issues.apache.org/jira/browse/OAK-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087360#comment-16087360
]
Valentin Olteanu commented on OAK-6452:
---
[~frm], [~mduerig], can you please take a look?
> IllegalStateException: too much data for a segment during oak-upgrade from
> segment to segment-tar
> -
>
> Key: OAK-6452
> URL: https://issues.apache.org/jira/browse/OAK-6452
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, upgrade
>Affects Versions: 1.7.3
>Reporter: Valentin Olteanu
>Priority: Critical
> Fix For: 1.7.6
>
>
> During the migration of a big repo from the {{old-segment}} format to
> {{segment-tar}} using {{oak-upgrade-1.7.3}}, I've got the following error:
> {code}
> 14.07.2017 09:05:51.920 [main] *INFO*
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade - Copying node
> #89333: /oak:index/uuid/:index/a9f9a3ed-6183-4e9e-9480-1b4fd196a829
> 14.07.2017 10:00:27.957 [TarMK flush
> [extracted/crx-quickstart/repository-oak-upgrade/segmentstore]] *ERROR*
> org.apache.jackrabbit.oak.segment.file.SafeRunnable - Uncaught exception in
> TarMK flush [extracted/crx-quickstart/repository-oak-upgrade/segmentstore]
> java.lang.IllegalStateException: too much data for a segment
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:322)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush(SegmentBufferWriterPool.java:142)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.DefaultSegmentWriter.flush(DefaultSegmentWriter.java:138)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:345)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:342)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(TarRevisions.java:213)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(TarRevisions.java:201)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.flush(FileStore.java:342)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:242)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67)
> ~[oak-upgrade-1.7.3.jar:1.7.3]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_131]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_131]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_131]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> 14.07.2017 10:00:28.448 [main] *INFO*
> org.apache.jackrabbit.oak.segment.file.FileStore - TarMK closed:
> extracted/crx-quickstart/repository-oak-upgrade/segmentstore
> 14.07.2017 10:00:28.658 [main] *INFO*
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore - TarMK closed:
> extracted/crx-quickstart/repository/segmentstore
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at
> org.apache.jackrabbit.oak.segment.RecordType.values(RecordType.java:24)
> at
> org.apache.jackrabbit.oak.segment.ImmutableRecordNumbers$1$1.getType(ImmutableRecordNumbers.java:86)
> at
> org.apache.jackrabbit.oak.segment.Segment.forEachRecord(Segment.java:703)
> at
> org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readBinaryReferences(AbstractFileStore.java:277)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(FileStore.java:511)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:356)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriter.prepare(SegmentBufferWriter.java:423)
> at
> org.apache.jackrabbit.oak.segment.RecordWriters$RecordWriter.write(RecordWriters.java:70)
> at
>
[jira] [Created] (OAK-6452) IllegalStateException: too much data for a segment during oak-upgrade from segment to segment-tar
Valentin Olteanu created OAK-6452:
-
Summary: IllegalStateException: too much data for a segment during
oak-upgrade from segment to segment-tar
Key: OAK-6452
URL: https://issues.apache.org/jira/browse/OAK-6452
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-tar, upgrade
Affects Versions: 1.7.3
Reporter: Valentin Olteanu
Priority: Critical
Fix For: 1.7.6
During the migration of a big repo from the {{old-segment}} format to
{{segment-tar}} using {{oak-upgrade-1.7.3}}, I've got the following error:
{code}
14.07.2017 09:05:51.920 [main] *INFO*
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade - Copying node
#89333: /oak:index/uuid/:index/a9f9a3ed-6183-4e9e-9480-1b4fd196a829
14.07.2017 10:00:27.957 [TarMK flush
[extracted/crx-quickstart/repository-oak-upgrade/segmentstore]] *ERROR*
org.apache.jackrabbit.oak.segment.file.SafeRunnable - Uncaught exception in
TarMK flush [extracted/crx-quickstart/repository-oak-upgrade/segmentstore]
java.lang.IllegalStateException: too much data for a segment
at
org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:322)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush(SegmentBufferWriterPool.java:142)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.DefaultSegmentWriter.flush(DefaultSegmentWriter.java:138)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:345)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:342)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(TarRevisions.java:213)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(TarRevisions.java:201)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.FileStore.flush(FileStore.java:342)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:242)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67)
~[oak-upgrade-1.7.3.jar:1.7.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_131]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
14.07.2017 10:00:28.448 [main] *INFO*
org.apache.jackrabbit.oak.segment.file.FileStore - TarMK closed:
extracted/crx-quickstart/repository-oak-upgrade/segmentstore
14.07.2017 10:00:28.658 [main] *INFO*
org.apache.jackrabbit.oak.plugins.segment.file.FileStore - TarMK closed:
extracted/crx-quickstart/repository/segmentstore
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at
org.apache.jackrabbit.oak.segment.RecordType.values(RecordType.java:24)
at
org.apache.jackrabbit.oak.segment.ImmutableRecordNumbers$1$1.getType(ImmutableRecordNumbers.java:86)
at
org.apache.jackrabbit.oak.segment.Segment.forEachRecord(Segment.java:703)
at
org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readBinaryReferences(AbstractFileStore.java:277)
at
org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(FileStore.java:511)
at
org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush(SegmentBufferWriter.java:356)
at
org.apache.jackrabbit.oak.segment.SegmentBufferWriter.prepare(SegmentBufferWriter.java:423)
at
org.apache.jackrabbit.oak.segment.RecordWriters$RecordWriter.write(RecordWriters.java:70)
at
org.apache.jackrabbit.oak.segment.DefaultSegmentWriter$SegmentWriteOperation.writeValueRecord(DefaultSegmentWriter.java:499)
at
org.apache.jackrabbit.oak.segment.DefaultSegmentWriter$SegmentWriteOperation.writeString(DefaultSegmentWriter.java:518)
at
org.apache.jackrabbit.oak.segment.DefaultSegmentWriter$SegmentWriteOperation.writeMap(DefaultSegmentWriter.java:324)
at
org.apache.jackrabbit.oak.segment.DefaultSegmentWriter$SegmentWriteOperation.writeNodeUncached(DefaultSegmentWriter.java:870)
at
[jira] [Commented] (OAK-4309) Align property labels and descriptions in SegmentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751710#comment-15751710
]
Valentin Olteanu commented on OAK-4309:
---
[~mduerig], as a general note, should we specify the default values (hardcoded)
in the descriptions so one can quickly see which values are changed when
looking at the console page?
For the rest, I have some individual comments:
bq. +description = "TarMK mode (64 for memory mapped file access,
32 for normal file access)."
Any hint on when to use one or the other?
bq. +description = "Cache size for storing most recently used
segments in megabytes."
This is valid for all the cache properties: specify that increasing this will
cause higher heap consumption so they need to ensure memory availability
bq. +label = "Disable estimation phase",
I would completely remove this option and specify that setting
{{compaction.sizeDeltaEstimation}} to 0 will skip estimation. Should I open a
separate issue for this?
{quote}
+description = "Boolean value indicating that a custom BlobStore is
used for storing " +
+"large binary values."
{quote}
Is it {{BlobStore}} or {{DataStore}}? Should we mention that an additional
config file for the specific DataStore is required ?
{quote}
+description = "Directory for storing repository backups. Defaults
to 'segmentstore-backup' " +
+"subdirectory under 'repository.home'."
{quote}
Does this accept relative paths? If yes, relative to what?
Also, something a little outside of the scope of this issue: I would love to
see the names of the properties also aligned, e.g. instead of
{{pauseCompaction}} have {{compaction.pause}} etc. Yet, this might introduce
even more backward non-compatibility...
> Align property labels and descriptions in SegmentNodeStoreService
> -
>
> Key: OAK-4309
> URL: https://issues.apache.org/jira/browse/OAK-4309
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: production
> Fix For: 1.5.16, 1.6
>
>
> We need to align / improve the labels and descriptions in
> {{SegmentNodeStoreService}} to match their actual purpose. At the same time I
> would opt for changing "compaction" to "revision gc" in all places where it
> is used synonymously for the latter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5225) Online compaction throttling
[
https://issues.apache.org/jira/browse/OAK-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729104#comment-15729104
]
Valentin Olteanu commented on OAK-5225:
---
[~alex.parvulescu], I haven't looked in depth to the patch, but have you
considered not adding sleeps during {{forceCompact}}? It wouldn't make sense to
delay compaction while the repository is locked.
> Online compaction throttling
>
>
> Key: OAK-5225
> URL: https://issues.apache.org/jira/browse/OAK-5225
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.16
>
>
> This is for the throttling part of OAK-5158.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-5161) Improve logging of compaction cycles
[ https://issues.apache.org/jira/browse/OAK-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-5161: -- Attachment: OAK-5161.patch [~mduerig], [~frm] or [~alex.parvulescu]: could you please take a look at the attached patch and see if it's worth adding this? > Improve logging of compaction cycles > > > Key: OAK-5161 > URL: https://issues.apache.org/jira/browse/OAK-5161 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Valentin Olteanu >Priority: Minor > Attachments: OAK-5161.patch > > > It would be useful to have logs with more info related to the compaction > cycles, e.g. how much each takes, show if force compact was successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-5161) Improve logging of compaction cycles
[ https://issues.apache.org/jira/browse/OAK-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-5161: -- Priority: Minor (was: Major) > Improve logging of compaction cycles > > > Key: OAK-5161 > URL: https://issues.apache.org/jira/browse/OAK-5161 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Valentin Olteanu >Priority: Minor > > It would be useful to have logs with more info related to the compaction > cycles, e.g. how much each takes, show if force compact was successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-5161) Improve logging of compaction cycles
Valentin Olteanu created OAK-5161: - Summary: Improve logging of compaction cycles Key: OAK-5161 URL: https://issues.apache.org/jira/browse/OAK-5161 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-tar Reporter: Valentin Olteanu It would be useful to have logs with more info related to the compaction cycles, e.g. how much each takes, show if force compact was successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-5158) Online compaction progress monitor
[
https://issues.apache.org/jira/browse/OAK-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693482#comment-15693482
]
Valentin Olteanu commented on OAK-5158:
---
+1 to separating {{logCycle}} from {{sleepPerCycle}}. One might want to have a
less granular logging, but shorter sleep cycles.
> Online compaction progress monitor
> --
>
> Key: OAK-5158
> URL: https://issues.apache.org/jira/browse/OAK-5158
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.15
>
> Attachments: OAK-5158.patch
>
>
> Having an online compaction progress monitor open the door for providing a
> (fuzzy) percentage of completion. Similar to the offline progress tracker, it
> would be responsible for logging progress at a configurable interval but also
> providing {{current number of compacted nodes}} and {{estimated number of
> total nodes}}.
> I'd also like to add the possibility to sleep a few ms each cycle, this can
> introduce throttling options for online compaction in the case the impact is
> too great on the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5058) Improve GC estimation strategy based on both absolute size and relative percentage
[
https://issues.apache.org/jira/browse/OAK-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690305#comment-15690305
]
Valentin Olteanu commented on OAK-5058:
---
I think having two settings for the same thing is an overkill and introduces
unnecessary confusion and complication in determining when compaction will run.
The reason to have an estimation is to save cpu cycles by determining with a
*simple heuristic* if it's worth running compaction. This check should be easy
to understand and the default setting should work in most of the cases, but for
sure cannot meet all the needs, thus leaving the config option to the user.
To summarize the three options, here's a list of advantages and disadvantages
that I see:
1. Only absolute delta threshold
* PRO: ensures compaction will not run when the growth is very small,
independently of the repo size: e.g. an increase of 1GB shows that you can
potentially save max 1GB, which makes it easy to relate to exact disk size
* PRO: easy to understand and foresee
* CON: when the delta is set too high, the repo might grow too much, thus
taking longer to compact
2. Only relative delta threshold
* PRO: ensures compaction will also run on smaller repos which don't grow too
much
* PRO: easy to understand and foresee
* CON: might trigger a compaction that saves only a few MB in case of small
repos - in this case compaction should anyway be fast, so it's not that bad
3. Both relative and absolute delta threshold
* PRO: (potentially) finer control of when to trigger compaction
* CON: confusing and hard to foresee
* CON: hard to get the two values correlated + needs exhaustive documentation
* CON: can lead to weird cases, e.g. if one wants to rely only on one of the
thresholds.
Personally I would go for the relative check, but maybe more input on PROs /
CONs should be considered.
As a side note, I think we should remove {{compaction.disableEstimation}} and
keep it simple, as it's the case for {{compaction.memoryThreshold}}:
{code}
Value represents a percentage so an input between 0 and 100 is expected.
Setting this to 0 will disable the check.
{code}
> Improve GC estimation strategy based on both absolute size and relative
> percentage
> --
>
> Key: OAK-5058
> URL: https://issues.apache.org/jira/browse/OAK-5058
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Affects Versions: 1.5.12
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Minor
> Fix For: 1.6, 1.5.15
>
> Attachments: OAK-5058-01.patch
>
>
> A better way of deciding whether GC should run or not might be by looking at
> the numbers computed in {{SizeDeltaGcEstimation}} from both an absolute size
> and relative percentage point of view. For example it would make sense to
> run compaction only if at least one criterion is met: "run if there is > 50%
> increase or more than 10GB".
> Since the absolute threshold is already implemented (see
> {{SegmentGCOptions.SIZE_DELTA_ESTIMATION_DEFAULT}}), it would be nice to add
> also something like {{SegmentGCOptions.SIZE_PERCENTAGE_ESTIMATION_DEFAULT}}
> and use it in making the decision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670839#comment-15670839
]
Valentin Olteanu commented on OAK-5117:
---
I agree, it's simpler and cleaner.
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117-v2.patch, OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5069) Backup fails when called from RepositoryManagementMBean#startBackup
[
https://issues.apache.org/jira/browse/OAK-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647123#comment-15647123
]
Valentin Olteanu commented on OAK-5069:
---
I would say this is more a parameter to `startBackup` and `startRestore` than a
global config (one might want to call `startBackup` at different times for
different destinations - think about hourly/daily/monthly backups).
In any case, I agree it should have a sensitive default, but I would go for the
more homogenous `segmentstore-backup` :)
Correct me if I'm wrong but I think this is an incremental backup. If not,
maybe the default should include a timestamp.
> Backup fails when called from RepositoryManagementMBean#startBackup
> ---
>
> Key: OAK-5069
> URL: https://issues.apache.org/jira/browse/OAK-5069
> Project: Jackrabbit Oak
> Issue Type: Bug
>Affects Versions: 1.5.12
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
> Fix For: 1.5.14
>
>
> When calling {{RepositoryManagementMBean.startBackup}}, the operation fails
> with the following stacktrace:
> {code:java}
> 04.11.2016 13:12:56.733 *ERROR* [qtp2039314079-250]
> org.apache.jackrabbit.oak.management.ManagementOperation Backup failed
> java.lang.IllegalStateException: /repository/segmentstore is in use by
> another store.
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:168)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:304)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupImpl.backup(FileStoreBackupImpl.java:65)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupRestoreImpl$1.call(FileStoreBackupRestoreImpl.java:102)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupRestoreImpl$1.call(FileStoreBackupRestoreImpl.java:97)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1062)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:166)
> ... 8 common frames omitted
> {code}
> /cc [~mduerig]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-5069) Backup fails when called from RepositoryManagementMBean#startBackup
[
https://issues.apache.org/jira/browse/OAK-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647123#comment-15647123
]
Valentin Olteanu edited comment on OAK-5069 at 11/8/16 10:05 AM:
-
I would say this is more a parameter to {{startBackup}} and {{startRestore}}
than a global config (one might want to call {{startBackup}} at different times
for different destinations - think about hourly/daily/monthly backups).
In any case, I agree it should have a sensitive default, but I would go for the
more homogenous {{segmentstore-backup}} :)
Correct me if I'm wrong but I think this is an incremental backup. If not,
maybe the default should include a timestamp.
was (Author: volteanu):
I would say this is more a parameter to `startBackup` and `startRestore` than a
global config (one might want to call `startBackup` at different times for
different destinations - think about hourly/daily/monthly backups).
In any case, I agree it should have a sensitive default, but I would go for the
more homogenous `segmentstore-backup` :)
Correct me if I'm wrong but I think this is an incremental backup. If not,
maybe the default should include a timestamp.
> Backup fails when called from RepositoryManagementMBean#startBackup
> ---
>
> Key: OAK-5069
> URL: https://issues.apache.org/jira/browse/OAK-5069
> Project: Jackrabbit Oak
> Issue Type: Bug
>Affects Versions: 1.5.12
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
> Fix For: 1.5.14
>
>
> When calling {{RepositoryManagementMBean.startBackup}}, the operation fails
> with the following stacktrace:
> {code:java}
> 04.11.2016 13:12:56.733 *ERROR* [qtp2039314079-250]
> org.apache.jackrabbit.oak.management.ManagementOperation Backup failed
> java.lang.IllegalStateException: /repository/segmentstore is in use by
> another store.
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:168)
> at
> org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.build(FileStoreBuilder.java:304)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupImpl.backup(FileStoreBackupImpl.java:65)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupRestoreImpl$1.call(FileStoreBackupRestoreImpl.java:102)
> at
> org.apache.jackrabbit.oak.backup.impl.FileStoreBackupRestoreImpl$1.call(FileStoreBackupRestoreImpl.java:97)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1062)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.(FileStore.java:166)
> ... 8 common frames omitted
> {code}
> /cc [~mduerig]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5000) Upgrade Oak Segment Tar dependency to 0.0.18
[ https://issues.apache.org/jira/browse/OAK-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605344#comment-15605344 ] Valentin Olteanu commented on OAK-5000: --- +1 > Upgrade Oak Segment Tar dependency to 0.0.18 > > > Key: OAK-5000 > URL: https://issues.apache.org/jira/browse/OAK-5000 > Project: Jackrabbit Oak > Issue Type: Task > Components: parent >Reporter: Davide Giannella >Assignee: Michael Dürig >Priority: Blocker > Fix For: 1.5.13 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4617) Align SegmentRevisionGC MBean with new generation based GC
[
https://issues.apache.org/jira/browse/OAK-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584895#comment-15584895
]
Valentin Olteanu commented on OAK-4617:
---
Regarding this, the current "Status" is showing the last log info. This is
useful, but it would also be interesting to expose the real-time status. For
monitoring it would be useful to know exactly in which phase we are, e.g. a
field showing on of the following:
- idle
- estimation
- compaction
- compaction-retry-1
- compaction-retry-2
- compaction-forcecompact
- cleanup
Do you think this is possible/useful to add it to this MBean?
> Align SegmentRevisionGC MBean with new generation based GC
> --
>
> Key: OAK-4617
> URL: https://issues.apache.org/jira/browse/OAK-4617
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: production
> Fix For: Segment Tar 0.0.16
>
>
> The {{SegmentRevisionGC}} MBean still dates back to the old {{oak-segment}}.
> We need to review its endpoints and only keep those that make sense for
> {{oak-segment-tar}}, adapt the others as necessary any add further
> functionality as required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4901) Improve SizeDeltaGcEstimation logging
[
https://issues.apache.org/jira/browse/OAK-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552536#comment-15552536
]
Valentin Olteanu edited comment on OAK-4901 at 10/7/16 7:52 AM:
[~alex.parvulescu] could you please review the attached patch? Thanks!
PS: I've just realized the patch has an unwanted change in {{pom.xml}} which
should be ignored.
was (Author: volteanu):
[~alex.parvulescu] could you please review the attached patch? Thanks!
> Improve SizeDeltaGcEstimation logging
> -
>
> Key: OAK-4901
> URL: https://issues.apache.org/jira/browse/OAK-4901
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.14
>Reporter: Valentin Olteanu
>Priority: Minor
> Attachments: OAK-4901.patch
>
>
> The current message logged after estimation is not very clear for someone
> that does not know how it works:
> {code}
> 06.10.2016 18:53:07.819 *INFO* [TarMK revision gc
> [/Users/volteanu/workspace/test/author/crx-quickstart/repository/segmentstore]]
> org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #1: estimation
> completed in 7.362 ms (7 ms). Size delta is 0% or 880.3 MB/880.9 MB
> (880258560/880892416 bytes), so skipping compaction for now
> {code}
> I think it should also list the delta in absolute value and the threshold it
> is compared against.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4901) Improve SizeDeltaGcEstimation logging
[
https://issues.apache.org/jira/browse/OAK-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-4901:
--
Attachment: OAK-4901.patch
[~alex.parvulescu] could you please review the attached patch? Thanks!
> Improve SizeDeltaGcEstimation logging
> -
>
> Key: OAK-4901
> URL: https://issues.apache.org/jira/browse/OAK-4901
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.14
>Reporter: Valentin Olteanu
>Priority: Minor
> Attachments: OAK-4901.patch
>
>
> The current message logged after estimation is not very clear for someone
> that does not know how it works:
> {code}
> 06.10.2016 18:53:07.819 *INFO* [TarMK revision gc
> [/Users/volteanu/workspace/test/author/crx-quickstart/repository/segmentstore]]
> org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #1: estimation
> completed in 7.362 ms (7 ms). Size delta is 0% or 880.3 MB/880.9 MB
> (880258560/880892416 bytes), so skipping compaction for now
> {code}
> I think it should also list the delta in absolute value and the threshold it
> is compared against.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4901) Improve SizeDeltaGcEstimation logging
Valentin Olteanu created OAK-4901:
-
Summary: Improve SizeDeltaGcEstimation logging
Key: OAK-4901
URL: https://issues.apache.org/jira/browse/OAK-4901
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segment-tar
Affects Versions: Segment Tar 0.0.14
Reporter: Valentin Olteanu
Priority: Minor
The current message logged after estimation is not very clear for someone that
does not know how it works:
{code}
06.10.2016 18:53:07.819 *INFO* [TarMK revision gc
[/Users/volteanu/workspace/test/author/crx-quickstart/repository/segmentstore]]
org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #1: estimation
completed in 7.362 ms (7 ms). Size delta is 0% or 880.3 MB/880.9 MB
(880258560/880892416 bytes), so skipping compaction for now
{code}
I think it should also list the delta in absolute value and the threshold it is
compared against.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3695) Expose ratio between waste and real data
[
https://issues.apache.org/jira/browse/OAK-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432339#comment-15432339
]
Valentin Olteanu commented on OAK-3695:
---
Indeed, this info is interesting especially in the context of compaction.
Customers might want to have some statistics about it in order to decide their
maintenance schedule, but since it proves to be an expensive operation, I am
fine to have it as a functionality of {{oak-run}} (especially that it can now
run in read-only mode on open repositories, right?).
Besides, for development and testing, this would be priceless.
> Expose ratio between waste and real data
>
>
> Key: OAK-3695
> URL: https://issues.apache.org/jira/browse/OAK-3695
> Project: Jackrabbit Oak
> Issue Type: Story
> Components: segment-tar
>Reporter: Valentin Olteanu
> Labels: gc
>
> As a user, I want to know the ratio (or precise absolute values) between
> waste and real data on TarMK, so that I can decide if Revision GC needs to be
> run. The measurement has to be done on a running repository and without
> impacting the performance.
> This would also help measure the efficiency of Revision GC and see the effect
> of improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4555) Deadlock in CompactionStats.writeNode during online compaction
[ https://issues.apache.org/jira/browse/OAK-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-4555: -- Attachment: stacktrace.log > Deadlock in CompactionStats.writeNode during online compaction > -- > > Key: OAK-4555 > URL: https://issues.apache.org/jira/browse/OAK-4555 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: Segment Tar 0.0.2 >Reporter: Valentin Olteanu > Attachments: stacktrace.log > > > While running online revision cleanup under high load, the instance reached a > deadlock (or possible infinite loop). > Full thread dump is attached, but it seems that: > 1. Thread-21 tries to commit -> CompactionStats.writeNode -> locked by > compaction thread > 2. TarMK compaction thread tries to compact -> print statistics -> infinite > loop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4555) Deadlock in CompactionStats.writeNode during online compaction
[ https://issues.apache.org/jira/browse/OAK-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375113#comment-15375113 ] Valentin Olteanu commented on OAK-4555: --- [~frm], this is the issue I've showed you live :) Please let me know if you need more info. > Deadlock in CompactionStats.writeNode during online compaction > -- > > Key: OAK-4555 > URL: https://issues.apache.org/jira/browse/OAK-4555 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: Segment Tar 0.0.2 >Reporter: Valentin Olteanu > Attachments: stacktrace.log > > > While running online revision cleanup under high load, the instance reached a > deadlock (or possible infinite loop). > Full thread dump is attached, but it seems that: > 1. Thread-21 tries to commit -> CompactionStats.writeNode -> locked by > compaction thread > 2. TarMK compaction thread tries to compact -> print statistics -> infinite > loop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4555) Deadlock in CompactionStats.writeNode during online compaction
Valentin Olteanu created OAK-4555: - Summary: Deadlock in CompactionStats.writeNode during online compaction Key: OAK-4555 URL: https://issues.apache.org/jira/browse/OAK-4555 Project: Jackrabbit Oak Issue Type: Bug Components: segment-tar Affects Versions: Segment Tar 0.0.2 Reporter: Valentin Olteanu Attachments: stacktrace.log While running online revision cleanup under high load, the instance reached a deadlock (or possible infinite loop). Full thread dump is attached, but it seems that: 1. Thread-21 tries to commit -> CompactionStats.writeNode -> locked by compaction thread 2. TarMK compaction thread tries to compact -> print statistics -> infinite loop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4400) Correlate index with the index definition used to build it
[ https://issues.apache.org/jira/browse/OAK-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298468#comment-15298468 ] Valentin Olteanu commented on OAK-4400: --- [~tmueller], we have discussed a little about this offline, so I created the issue to continue the idea. I'm not an indexing expert and I might have imagined too much, so please add your input here and let me know what you think about this. > Correlate index with the index definition used to build it > -- > > Key: OAK-4400 > URL: https://issues.apache.org/jira/browse/OAK-4400 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, query >Affects Versions: 1.4 >Reporter: Valentin Olteanu > > Currently, if the definition of an index is changed without reindexing, it > will get in an "inconsistent" state. > Of course, the reindexing is usually necessary, but it would be useful to > know with which definition the index was built. This could increase the > visibility of the indexing state and help debugging issues related to it. > Some questions this improvement should respond to: > # What is the definition of the index when the (re)indexing was triggered? > # Are there any changes in the definition since the trigger? Which? > I can imagine a solution built by "versioning" the definition nodes > (oak:QueryIndexDefinition). When the reindex is triggered, a new version of > the node is created and the indexer stores a reference to it. > This would also allow the indexer to keep using the same definition until a > new reindex, even if changes are made meanwhile (i.e. use a fixed version > instead of the latest definition). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4400) Correlate index with the index definition used to build it
Valentin Olteanu created OAK-4400: - Summary: Correlate index with the index definition used to build it Key: OAK-4400 URL: https://issues.apache.org/jira/browse/OAK-4400 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene, query Affects Versions: 1.4 Reporter: Valentin Olteanu Currently, if the definition of an index is changed without reindexing, it will get in an "inconsistent" state. Of course, the reindexing is usually necessary, but it would be useful to know with which definition the index was built. This could increase the visibility of the indexing state and help debugging issues related to it. Some questions this improvement should respond to: # What is the definition of the index when the (re)indexing was triggered? # Are there any changes in the definition since the trigger? Which? I can imagine a solution built by "versioning" the definition nodes (oak:QueryIndexDefinition). When the reindex is triggered, a new version of the node is created and the indexer stores a reference to it. This would also allow the indexer to keep using the same definition until a new reindex, even if changes are made meanwhile (i.e. use a fixed version instead of the latest definition). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4211) FileAccess.Mapped leaks file channels
[
https://issues.apache.org/jira/browse/OAK-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245421#comment-15245421
]
Valentin Olteanu commented on OAK-4211:
---
[~mduerig], is the leak triggered only in the case of OAK-3348?
> FileAccess.Mapped leaks file channels
> --
>
> Key: OAK-4211
> URL: https://issues.apache.org/jira/browse/OAK-4211
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
> Fix For: 1.6
>
>
> {{FileAccess.Mapped}} seems to leak {{FileChannnel}} instances. The file
> channels are acquired in the constructor but never release.
> FYI [~alex.parvulescu], [~frm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3814) Collect and expose important runtime statistics across various part of Oak
[ https://issues.apache.org/jira/browse/OAK-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082844#comment-15082844 ] Valentin Olteanu edited comment on OAK-3814 at 1/5/16 11:02 AM: Indeed, this epic has a broader scope but includes OAK-3693. We can move the issues here and close it. was (Author: volteanu): Indeed, this epic have a broader scope but includes OAK-3693. We can move the issues here and close it. > Collect and expose important runtime statistics across various part of Oak > -- > > Key: OAK-3814 > URL: https://issues.apache.org/jira/browse/OAK-3814 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.4 > > > Aim of this epic is to improve the collection of runtime statistics across > various sub systems in Oak. This would enable following benefits > # Better insight into operational aspects of Oak > # Better integration with external monitoring system based on JMX > # Based on stats we can decide what needs to be done to make Oak perform > better i.e. based on actual usage data -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-3694:
--
Description:
As a user, I want to know when a node that I've created has been indexed so
that I can start using it.
Generalization: As a user, I want to know when everything created before
timestamp T0 has been indexed.
Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
returns {{true}} if everything created before {{timestamp}} has been indexed
(by all indexers).
Current options:
# check IndexStatsMBean for Start and Done to determine if a cycle has started
after adding the node and has finished. Issue: there can be multiple async
indexers. OAK-3606 proposes a solution.
# add a node and search for it until it is retrieved. Issue: must add different
nodes for different indexers and wait for all.
These options are not precise (give false positives) and are not resilient to
changes in indexing strategy (e.g. adding one more indexer breaks the checks).
was:
As a user, I want to know when a node that I've created has been indexed so
that I can start using it.
Generalization: As a user, I want to know when everything created before
timestamp T0 has been indexed.
Current options: (1) check
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044760#comment-15044760
]
Valentin Olteanu commented on OAK-3694:
---
[~chetanm], as discussed during the oakathon, I've updated the story with the
requirements. Thierry had proposed a solution on OAK-3606, but please comment
with any idea that you have on this.
It would be really helpful to have this done in a reliable way.
Thanks!
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[ https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-3694: -- Description: As a user, I want to know when a node that I've created has been indexed so that I can start using it. Generalization: As a user, I want to know when everything created before timestamp T0 has been indexed. Current options: (1) check was: As a user, I want to know when a node that I've created has been indexed so that I can start using it. Generalization: As a user, I want to know when everything created before timestamp T0 has been indexed. > As a user, I want to know when a node that I've created has been indexed. > - > > Key: OAK-3694 > URL: https://issues.apache.org/jira/browse/OAK-3694 > Project: Jackrabbit Oak > Issue Type: Story >Reporter: Valentin Olteanu > > As a user, I want to know when a node that I've created has been indexed so > that I can start using it. > Generalization: As a user, I want to know when everything created before > timestamp T0 has been indexed. > Current options: (1) check -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3693) Expose the internal state of the repository through indicators and checks
Valentin Olteanu created OAK-3693: - Summary: Expose the internal state of the repository through indicators and checks Key: OAK-3693 URL: https://issues.apache.org/jira/browse/OAK-3693 Project: Jackrabbit Oak Issue Type: Epic Reporter: Valentin Olteanu This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-3693: -- Description: This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. Areas to improve: * Async Indexing * Revision GC * Topology changes was:This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Valentin Olteanu > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC > * Topology changes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
Valentin Olteanu created OAK-3694: - Summary: As a user, I want to know when a node that I've created has been indexed. Key: OAK-3694 URL: https://issues.apache.org/jira/browse/OAK-3694 Project: Jackrabbit Oak Issue Type: Story Reporter: Valentin Olteanu As a user, I want to know when a node that I've created has been indexed so that I can start using it. Generalization: As a user, I want to know when everything created before timestamp T0 has been indexed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3695) As a user, I want to know the ratio between waste and real data
Valentin Olteanu created OAK-3695: - Summary: As a user, I want to know the ratio between waste and real data Key: OAK-3695 URL: https://issues.apache.org/jira/browse/OAK-3695 Project: Jackrabbit Oak Issue Type: Story Reporter: Valentin Olteanu As a user, I want to know the ratio (or precise absolute values) between waste and real data on TarMK, so that I can decide if Revision GC needs to be run. The measurement has to be done on a running repository and without impacting the performance. This would also help measure the efficiency of Revision GC and see the effect of improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-3693: -- Description: This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. Areas to improve: * Async Indexing * Revision GC was: This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. Areas to improve: * Async Indexing * Revision GC * Topology changes * Workflows * Replication > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Valentin Olteanu > Labels: production > Fix For: 1.4 > > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-3693: -- Description: This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. Areas to improve: * Async Indexing * Revision GC * Topology changes * Workflows * Replication was: This container groups all the issues related to areas where we could improve the monitoring of an OAK repository. This can be achieved by exposing different indicators of the internal state and adding checks for certain properties. Areas to improve: * Async Indexing * Revision GC * Topology changes > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Valentin Olteanu > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC > * Topology changes > * Workflows > * Replication -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3506) Uniformization of compaction log messages
Valentin Olteanu created OAK-3506:
-
Summary: Uniformization of compaction log messages
Key: OAK-3506
URL: https://issues.apache.org/jira/browse/OAK-3506
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Reporter: Valentin Olteanu
The logs generated during different phases of tar garbage collection
(compaction) are currently quite heterogenous and difficult to grep/parse.
I propose with the attached patch to uniformize these logs, changing the
following:
# all logs start with the prefix {{TarMK GargabeCollection \{\}#:}}
# different phases of garbage collection are easier to identify by the first
word after prefix, e.g. estimation, compaction, cleanup
# all values are also printed in a standard unit, with the following format:
{{ ()}}. This makes extraction of
information much easier.
# messages corresponding to the same cycle (run) can be grouped by including
the runId in the prefix.
Note1: I don't have enough visibility, but the changes might impact any system
relying on the old format. Yet, I've seen they have changed before so this
might not be a real concern.
Note2: the runId is implemented as a static variable, which is reset every time
the class is reloaded (e.g. at restart), so it is unique only during one run.
Below you can find an excerpt of old logs and new logs to compare:
NEW:
{code}
12.10.2015 16:11:56.705 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: started
12.10.2015 16:11:56.707 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: estimation started
12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: estimation completed in 2.569 s (2567 ms). Gain is 16% or
1.1 GB/1.3 GB (1062364160/1269737472 bytes), so running compaction
12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: compaction started,
strategy=CompactionStrategy{paused=false, cloneBinaries=false,
cleanupType=CLEAN_OLD, olderThan=3600, memoryThreshold=5,
persistedCompactionMap=true, retryCount=5, forceAfterFail=true,
compactionStart=1444659116706, offlineCompaction=false}
12.10.2015 16:12:05.839 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.Compactor Finished compaction: 420022
nodes, 772259 properties, 20544 binaries.
12.10.2015 16:12:07.459 *INFO* [TarMK compaction thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: compaction completed in 8.184 s (8183 ms), after 0 cycles
12.10.2015 16:12:11.912 *INFO* [TarMK flush thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: cleanup started. Current repository size is 1.4 GB
(1368899584 bytes)
12.10.2015 16:12:12.368 *INFO* [TarMK flush thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: cleanup marking file for deletion: data8a.tar
12.10.2015 16:12:12.434 *INFO* [TarMK flush thread
[/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
GarbageCollection #1: cleanup completed in 522.8 ms (522 ms). Post cleanup size
[jira] [Updated] (OAK-3506) Uniformization of compaction log messages
[
https://issues.apache.org/jira/browse/OAK-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-3506:
--
Attachment: compaction_logs.git.patch
patch to be applied with: patch -p0 -i compaction_logs.git.patch
> Uniformization of compaction log messages
> -
>
> Key: OAK-3506
> URL: https://issues.apache.org/jira/browse/OAK-3506
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Valentin Olteanu
> Attachments: compaction_logs.git.patch
>
>
> The logs generated during different phases of tar garbage collection
> (compaction) are currently quite heterogenous and difficult to grep/parse.
> I propose with the attached patch to uniformize these logs, changing the
> following:
> # all logs start with the prefix {{TarMK GargabeCollection \{\}#:}}
> # different phases of garbage collection are easier to identify by the first
> word after prefix, e.g. estimation, compaction, cleanup
> # all values are also printed in a standard unit, with the following format:
> {{ ()}}. This makes extraction of
> information much easier.
> # messages corresponding to the same cycle (run) can be grouped by including
> the runId in the prefix.
> Note1: I don't have enough visibility, but the changes might impact any
> system relying on the old format. Yet, I've seen they have changed before so
> this might not be a real concern.
> Note2: the runId is implemented as a static variable, which is reset every
> time the class is reloaded (e.g. at restart), so it is unique only during one
> run.
> Below you can find an excerpt of old logs and new logs to compare:
> NEW:
> {code}
> 12.10.2015 16:11:56.705 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: started
> 12.10.2015 16:11:56.707 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: estimation started
> 12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: estimation completed in 2.569 s (2567 ms). Gain is 16%
> or 1.1 GB/1.3 GB (1062364160/1269737472 bytes), so running compaction
> 12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: compaction started,
> strategy=CompactionStrategy{paused=false, cloneBinaries=false,
> cleanupType=CLEAN_OLD, olderThan=3600, memoryThreshold=5,
> persistedCompactionMap=true, retryCount=5, forceAfterFail=true,
> compactionStart=1444659116706, offlineCompaction=false}
> 12.10.2015 16:12:05.839 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.Compactor Finished compaction:
> 420022 nodes, 772259 properties, 20544 binaries.
> 12.10.2015 16:12:07.459 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: compaction completed in 8.184 s (8183 ms), after 0
> cycles
> 12.10.2015 16:12:11.912 *INFO* [TarMK flush thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: cleanup started. Current repository size is 1.4 GB
> (1368899584 bytes)
> 12.10.2015 16:12:12.368 *INFO* [TarMK flush thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore
[jira] [Commented] (OAK-3506) Uniformization of compaction log messages
[
https://issues.apache.org/jira/browse/OAK-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953208#comment-14953208
]
Valentin Olteanu commented on OAK-3506:
---
/cc [~alex.parvulescu], [~mduerig], [~frm] could you please have a look and
tell if this is a good idea?
> Uniformization of compaction log messages
> -
>
> Key: OAK-3506
> URL: https://issues.apache.org/jira/browse/OAK-3506
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Valentin Olteanu
> Attachments: compaction_logs.git.patch
>
>
> The logs generated during different phases of tar garbage collection
> (compaction) are currently quite heterogenous and difficult to grep/parse.
> I propose with the attached patch to uniformize these logs, changing the
> following:
> # all logs start with the prefix {{TarMK GargabeCollection \{\}#:}}
> # different phases of garbage collection are easier to identify by the first
> word after prefix, e.g. estimation, compaction, cleanup
> # all values are also printed in a standard unit, with the following format:
> {{ ()}}. This makes extraction of
> information much easier.
> # messages corresponding to the same cycle (run) can be grouped by including
> the runId in the prefix.
> Note1: I don't have enough visibility, but the changes might impact any
> system relying on the old format. Yet, I've seen they have changed before so
> this might not be a real concern.
> Note2: the runId is implemented as a static variable, which is reset every
> time the class is reloaded (e.g. at restart), so it is unique only during one
> run.
> Below you can find an excerpt of old logs and new logs to compare:
> NEW:
> {code}
> 12.10.2015 16:11:56.705 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: started
> 12.10.2015 16:11:56.707 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: estimation started
> 12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: estimation completed in 2.569 s (2567 ms). Gain is 16%
> or 1.1 GB/1.3 GB (1062364160/1269737472 bytes), so running compaction
> 12.10.2015 16:11:59.275 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: compaction started,
> strategy=CompactionStrategy{paused=false, cloneBinaries=false,
> cleanupType=CLEAN_OLD, olderThan=3600, memoryThreshold=5,
> persistedCompactionMap=true, retryCount=5, forceAfterFail=true,
> compactionStart=1444659116706, offlineCompaction=false}
> 12.10.2015 16:12:05.839 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.Compactor Finished compaction:
> 420022 nodes, 772259 properties, 20544 binaries.
> 12.10.2015 16:12:07.459 *INFO* [TarMK compaction thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:11:56 CEST 2015, previous max duration 0ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: compaction completed in 8.184 s (8183 ms), after 0
> cycles
> 12.10.2015 16:12:11.912 *INFO* [TarMK flush thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK
> GarbageCollection #1: cleanup started. Current repository size is 1.4 GB
> (1368899584 bytes)
> 12.10.2015 16:12:12.368 *INFO* [TarMK flush thread
> [/Users/volteanu/workspace/test/qp/quickstart-author-4502/crx-quickstart/repository/segmentstore],
> active since Mon Oct 12 16:12:11 CEST 2015, previous max duration 10ms]
>
[jira] [Commented] (OAK-3273) ColdStandby JMX Status
[ https://issues.apache.org/jira/browse/OAK-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709242#comment-14709242 ] Valentin Olteanu commented on OAK-3273: --- [~alex.parvulescu], could you please take a look and review the patch I've created for this issue? ColdStandby JMX Status --- Key: OAK-3273 URL: https://issues.apache.org/jira/browse/OAK-3273 Project: Jackrabbit Oak Issue Type: Improvement Components: tarmk-standby Reporter: Valentin Olteanu Priority: Minor Attachments: OAK-3273.patch OAK-3113 introduced two fields in the ColdStandby MBean: SyncStartTimestamp and SyncEndTimestamp. This is much more useful than the old SecondsSinceLastSuccess, yet, there are situations in which it's hard to interpret them since they are updated independently: - it's impossible to correlate the start with the end - in case of fail, the start still reflects the failed cycle It would be even better if the two would be updated atomically, to reflect the start and end of the last successful cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3273) ColdStandby JMX Status
[ https://issues.apache.org/jira/browse/OAK-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Olteanu updated OAK-3273: -- Attachment: OAK-3273.patch ColdStandby JMX Status --- Key: OAK-3273 URL: https://issues.apache.org/jira/browse/OAK-3273 Project: Jackrabbit Oak Issue Type: Improvement Components: tarmk-standby Reporter: Valentin Olteanu Priority: Minor Attachments: OAK-3273.patch OAK-3113 introduced two fields in the ColdStandby MBean: SyncStartTimestamp and SyncEndTimestamp. This is much more useful than the old SecondsSinceLastSuccess, yet, there are situations in which it's hard to interpret them since they are updated independently: - it's impossible to correlate the start with the end - in case of fail, the start still reflects the failed cycle It would be even better if the two would be updated atomically, to reflect the start and end of the last successful cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3273) ColdStandby JMX Status
Valentin Olteanu created OAK-3273: - Summary: ColdStandby JMX Status Key: OAK-3273 URL: https://issues.apache.org/jira/browse/OAK-3273 Project: Jackrabbit Oak Issue Type: Improvement Components: tarmk-standby Reporter: Valentin Olteanu Priority: Minor OAK-3113 introduced two fields in the ColdStandby MBean: SyncStartTimestamp and SyncEndTimestamp. This is much more useful than the old SecondsSinceLastSuccess, yet, there are situations in which it's hard to interpret them since they are updated independently: - it's impossible to correlate the start with the end - in case of fail, the start still reflects the failed cycle It would be even better if the two would be updated atomically, to reflect the start and end of the last successful cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
