svn commit: r1857188 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md
Author: mduerig Date: Tue Apr 9 12:23:24 2019 New Revision: 1857188 URL: http://svn.apache.org/viewvc?rev=1857188=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection (WIP): - Cleanup before compaction in Oak 1.10 Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1857188=1857187=1857188=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Tue Apr 9 12:23:24 2019 @@ -107,5 +107,11 @@ Oak 1.6 removed the [`Compactor` class]( Oak 1.10 -* TODO: describe fixes, improvements and backports +With Oak 1.6 and Oak 1.8 it was observed that running compaction first increases the repository until cleanup runs and subsequently removes the generation that has become reclaimable. Oak 1.10 improved this aspect by running cleanup *before* compaction thus levelling out the bump in repository size cause by the compaction phase. + +The effort included a few refactorings making garbage collection more modular: +* [OAK-7377](https://issues.apache.org/jira/browse/OAK-7377) generalised the [garbage collector](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.10.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GarbageCollector.java#L46) allowing multiple implementations. +* [OAK-7440](https://issues.apache.org/jira/browse/OAK-7440), [OAK-7434](https://issues.apache.org/jira/browse/OAK-7434) and [OAK-7436](https://issues.apache.org/jira/browse/OAK-7436) factored estimation, compaction and cleanup into independent components. +* [OAK-7445](https://issues.apache.org/jira/browse/OAK-7445) introduced the new cleanup before compaction garbage collection strategy. +* [OAK-7550](https://issues.apache.org/jira/browse/OAK-7550) eventually set the cleanup before compaction strategy as the new default for Oak 1.10.
svn commit: r1857001 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md
Author: mduerig Date: Fri Apr 5 11:15:53 2019 New Revision: 1857001 URL: http://svn.apache.org/viewvc?rev=1857001=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection (WIP): - Describe usage of MemoryNodeBuilder in Compactor to handle many direct child nodes and track the stableId Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1857001=1857000=1857001=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Fri Apr 5 11:15:53 2019 @@ -101,8 +101,8 @@ This logic is captured in the respective > *Note:* Oak 1.8.0 had a > [bug](https://issues.apache.org/jira/browse/OAK-7132) in the implementation > of the reclamation mechanism described. The bug was fixed with Oak 1.8.1, > which is the version of Oak this section is referring to. -* TODO: describe usage of MemoryNodeBuilder and modCount in Compactor -* TODO: describe usage and implementation of stableIds in Compactor +### The Compactor strikes back +Oak 1.6 removed the [`Compactor` class](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/Compactor.java#L54) in favour of directly rewriting node states with the [`SegmentWriter`](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java#L85) solely relying on deduplication caches for deduplicating records. To implement sequential checkpoint rebasing and tail compaction Oak 1.8 reintroduced a new implementation of a [`Compactor` class](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L56). This implementation has to deal with two additional requirements compared to the previous implementation: tracking and assigning stable ids and being able to cope with a large number of direct child nodes of a node. This is done by tracking cha nges with a [`MemoryNodeBuilder`](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L163) instead of using a `NodeBuilder` acquired through calling `NodeState.builder`. The new [`SegmentWriter.write`](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java#L130) method with an extra argument for the stable is then used to [write](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L175) compacted node states including their stable id. In addition the number of updates to the `MemoryNodeBuilder` are [tracked](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L154-L160) and an [intermediate node is written](http s://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L156) to avoid keeping to many updates in memory once an [update limit](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L62) is exceeded. Further updates are tracked in a fresh `MemoryNodeBuilder` instance that uses this [intermediate node as its base](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.8.1/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Compactor.java#L157). Oak 1.10
svn commit: r1856909 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md
Author: mduerig Date: Thu Apr 4 09:05:46 2019 New Revision: 1856909 URL: http://svn.apache.org/viewvc?rev=1856909=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection (WIP): - Describe checkpoint rebasing and tail compaction in Oak 1.8 Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1856909=1856908=1856909=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Thu Apr 4 09:05:46 2019 @@ -57,7 +57,7 @@ Oak 1.6 was the first version with worka ### Generational garbage collection Oak 1.6 changed the mechanism used to determine reclaimability of segments. Previous versions used reachability through the segment graph starting from a set of GC roots consisting of the segment containing the current head node state and all segments containing records currently referenced by the JVM (i.e. by open sessions). -Oak 1.6 introduced the concept of a [GC generation](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java#L380). GC generations are numbered starting at 0 and increasing with each run of OnRC. Each segment records the current GC generation from the time the segment was created in its [segment header](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java#L200-L203). The current GC generation of the repository is just the GC generation of the segment containing the current head state. The compactor [reads](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L845) the current GC generation of the repository and [rewrites](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment- tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L864) the head state using the next GC generation number for the segments created in the process. Once the compactor finished rewriting the current head state the newly created, compact head state is [atomically set](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L876) as the new head state of the repository, implicitly and atomically increasing the GC generation of the repository at the same time. +Oak 1.6 introduced the concept of a [GC generation](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java#L380). A GC generation is an integer starting at 0 and increasing with each run of OnRC. Each segment records the current GC generation from the time the segment was created in its [segment header](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java#L200-L203). The current GC generation of the repository is just the GC generation of the segment containing the current head state. The compactor [reads](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L845) the current GC generation of the repository and [rewrites](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segmen t-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L864) the head state using the next GC generation number for the segments created in the process. Once the compactor finished rewriting the current head state the newly created, compact head state is [atomically set](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L876) as the new head state of the repository, implicitly and atomically increasing the GC generation of the repository at the same time. In its default configuration the [cleanup](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L1055) phase retains all segments from the current GC generation and the previous one reclaiming all older segments. With the default daily OnRC execution this results in a minimal segment retention time of 24 hours. Sessions that are open at the point in time where OnRC runs will automatically
svn commit: r1856466 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md
Author: mduerig Date: Thu Mar 28 10:30:24 2019 New Revision: 1856466 URL: http://svn.apache.org/viewvc?rev=1856466=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection: - Improved cross referencing into GitHub - Paragraph about how reachability is used for bulk segments in Oak 1.6 to determine reclaimability for those Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1856466=1856465=1856466=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Thu Mar 28 10:30:24 2019 @@ -57,9 +57,11 @@ Oak 1.6 was the first version with worka ### Generational garbage collection Oak 1.6 changed the mechanism used to determine reclaimability of segments. Previous versions used reachability through the segment graph starting from a set of GC roots consisting of the segment containing the current head node state and all segments containing records currently referenced by the JVM (i.e. by open sessions). -Oak 1.6 introduced the concept of a GC generation. GC generations are numbered starting at 0 and increasing with each run of OnRC. Each segment records the current GC generation from the time the segment was created in its segment header. The current GC generation of the repository is just the GC generation of the segment containing the current head state. The compactor reads the current GC generation of the repository and rewrites the head state using the next GC generation number for the segments created in the process. Once the compactor finished rewriting the current head state the newly created, compact head state is atomically set as the new head state of the repository, implicitly and atomically increasing the GC generation of the repository at the same time. +Oak 1.6 introduced the concept of a [GC generation](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java#L380). GC generations are numbered starting at 0 and increasing with each run of OnRC. Each segment records the current GC generation from the time the segment was created in its [segment header](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java#L200-L203). The current GC generation of the repository is just the GC generation of the segment containing the current head state. The compactor [reads](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L845) the current GC generation of the repository and [rewrites](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment- tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L864) the head state using the next GC generation number for the segments created in the process. Once the compactor finished rewriting the current head state the newly created, compact head state is [atomically set](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L876) as the new head state of the repository, implicitly and atomically increasing the GC generation of the repository at the same time. -In its default configuration the cleanup phase retains all segments from the current GC generation and the previous one reclaiming all older segments. With the default daily OnRC execution this results in a minimal segment retention time of 24 hours. Sessions that are open at the point in time where OnRC runs will automatically [refresh](https://issues.apache.org/jira/browse/OAK-2407) at next access to reduce the risk for them to reference content from segments that were reclaimed. +In its default configuration the [cleanup](https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.6.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L1055) phase retains all segments from the current GC generation and the previous one reclaiming all older segments. With the default daily OnRC execution this results in a minimal segment retention time of 24 hours. Sessions that are open at the point in time where OnRC runs will automatically [refresh](https://issues.apache.org/jira/browse/OAK-2407) at next access to reduce the risk for them to reference content from segments that were reclaimed. + + Since [bulk segments](http://jackrabbit.apache.org/oak/docs
svn commit: r1856288 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md
Author: mduerig Date: Tue Mar 26 10:43:39 2019 New Revision: 1856288 URL: http://svn.apache.org/viewvc?rev=1856288=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection: description of compaction and cleanup in Oak 1.4 and before. Special mention of the potential for cycles in the segment graph Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1856288=1856287=1856288=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Tue Mar 26 10:43:39 2019 @@ -33,14 +33,19 @@ Online Revision Cleanup (OnRC) refers to * Cleanup: reclaimable segments are removed. Reclaimability is determined either by reachability through the segment graph or by the age of the segment depending on the version of Oak. -Oak 1.4 +Oak 1.0 - 1.4 +- Online Revision Garbage Collection did not work up to and including Oak 1.4 as it was not able to collect any garbage. In these version OnRC relied upon false premises on one hand and was further impacted by a bug [(OAK-3348)](https://issues.apache.org/jira/browse/OAK-3348) on the other hand. -The record graph of a repository grows very large very quickly. To avoid traversing large record graphs, the cleanup phase would instead operate on the segment graph induced by the record graph. That is, for any two segments s1 and s2, there would be an edge from s1 to s2 if and only if s1 contains a record that references a record in s2. By construction the segment graph contains far less vertices than the record graph. To speed up traversal it is pre-calculated and cached in the segment headers. While the segment graph is sufficiently small for efficient traversal, it is also extremely dense. In fact it turned out that its reflexive, transitive closure is the entire graph most of the time. The reason for this can be seen when looking at an example where a segment contains just a single reachable record and *n* unreachable records. In this case the single reachable record makes the segment reachable preventing it from being reclaimed along with the *n* non reachable records. To mak e matters worse, all segments referenced from this segment will also stay in the reachable set, although the single reachable record might not have any outgoing references at all. +The [compaction](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1045) phase uses an instance of the [`Compactor`](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/Compactor.java#L54) class for [rewriting](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1061) the current head state of the repository. The `Compactor` itself works by [comparing](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/Compactor.java#L160) a `before` state to an `after` state applying the differences to an `onto` state. In an initial pass the current head state of the repository is passed for the `after` state and an [empty node](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-core/src/main /java/org/apache/jackrabbit/oak/plugins/memory/EmptyNodeState.java#L37) is passed for both `before` and `onto`. Once the initial phase completes an attempt is made to [set](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1075) the repository's head state to the resulting node state via an atomic compare and set operation. This fails in the case when concurrent write operations to the repository changed its head state in the meanwhile. In that case a retry loop is entered where these additional changes are [compacted](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1082) on top of the previously compacted head state. After a configurable (default 5) numbers of retires a final attempt is made to [force compact](https://github.com/apache/jackrabbit-oak/blob/1.4/oak-segment/src/main/java/org/apache/jackrabbi t/oak/plugins/segment/file/FileStore.java#L1099) any remaining changes while blocking writes to the repository for a configurable (default 1 minute) time. Only if force compacting also fails is the compaction considered failed. -During online
svn commit: r1856038 - in /jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment: onrc-memoirs.md overview.md
Author: mduerig Date: Fri Mar 22 10:32:43 2019 New Revision: 1856038 URL: http://svn.apache.org/viewvc?rev=1856038=rev Log: OAK-301: Document Oak Memoirs in Garbage Collection (WIP) Added: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Added: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md?rev=1856038=auto == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md (added) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/onrc-memoirs.md Fri Mar 22 10:32:43 2019 @@ -0,0 +1,57 @@ +Memoirs in Garbage Collection += +This is a brief outline of the history of Online Revision Garbage Collection in Oak. By linking to further details where necessary this historical context helps making sense of the various bits of information that are scattered across Jira Issues, Wikis, source code etc. + +Refer to [Oak Segment Tar](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html) on the Jackrabbit Oak Wiki for a general overview of the segment store, its design, data structures and inner workings. + + +Background +-- +Online Revision Cleanup (OnRC) refers to a technique employed by the segment store to reclaim disk space that is no longer in use. The implementation is structured in three phases: + +* Estimation: a heuristic to determine whether enough garbage has accumulated to warrant running garbage collection at all. + +* Compaction: all records of the segment store's current head state are rewritten into a new, structurally equal head state. The records of the rewritten head state are compact within their segment as rewriting skips all records that are not reachable from the root node state. + +* Cleanup: reclaimable segments are removed. Reclaimability is determined either by reachability through the segment graph or by the age of the segment depending on the version of Oak. + + +Oak 1.4 +--- +Online Revision Garbage Collection did not work up to and including Oak 1.4 as it was not able to collect any garbage. In these version OnRC relied upon false premises on one hand and was further impacted by a bug [(OAK-3348)](https://issues.apache.org/jira/browse/OAK-3348) on the other hand. + +The record graph of a repository grows very large very quickly. To avoid traversing large record graphs, the cleanup phase would instead operate on the segment graph induced by the record graph. That is, for any two segments s1 and s2, there would be an edge from s1 to s2 if and only if s1 contains a record that references a record in s2. By construction the segment graph contains far less vertices than the record graph. To speed up traversal it is pre-calculated and cached in the segment headers. While the segment graph is sufficiently small for efficient traversal, it is also extremely dense. In fact it turned out that its reflexive, transitive closure is the entire graph most of the time. The reason for this can be seen when looking at an example where a segment contains just a single reachable record and *n* unreachable records. In this case the single reachable record makes the segment reachable preventing it from being reclaimed along with the *n* non reachable records. To mak e matters worse, all segments referenced from this segment will also stay in the reachable set, although the single reachable record might not have any outgoing references at all. + +During online compaction all records of the current head state of the repository are rewritten to form a new and compact representation of the repository's head in a separate set of segments. However, the presence of open sessions referencing older revisions for a while prevents those from being reclaimed. +In addition [(OAK-3348)](https://issues.apache.org/jira/browse/OAK-3348) could cause some of the segments of the compacted revision to still reference segments from the pre-compacted revision. Together with the segment graph being very dense this prevented almost any garbage collection from happening in Oak 1.4 and earlier. + +See also the annotated slides [Into the tar pit: a TarMK deep dive](https://adapt.to/2016/en/schedule/into-the-tar-pit--a-tarmk-deep-dive.html) for further illustrations on this topic. + + +Oak 1.6 +--- +Oak 1.6 was the first version with workable OnRC overcoming the problems with with open sessions keeping references to previous revisions and fixing [(OAK-3348)](https://issues.apache.org/jira/browse/OAK-3348). This required changes in the persistence format forcing existing customers to [migrate](https://helpx.adobe.com/experience-manager
svn commit: r1855590 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ main/java/org/apache/jackrabbit/oak/segment/scheduler/ test/java/org/apache/jackrabbit
Author: mduerig
Date: Fri Mar 15 13:35:08 2019
New Revision: 1855590
URL: http://svn.apache.org/viewvc?rev=1855590=rev
Log:
OAK-8094: JMX monitoring to detect commits carrying over from previous GC
generation can block other threads from committing
Expose gc generation of commit
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreMonitor.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStats.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java?rev=1855590=1855589=1855590=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Fri Mar 15 13:35:08 2019
@@ -29,9 +29,11 @@ import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import java.util.stream.Stream;
+import java.util.function.Supplier;
import com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
+import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
/**
@@ -43,7 +45,11 @@ import org.jetbrains.annotations.Nullabl
* currently waiting on the commit semaphore
*
*
- * This class delegates thread-safety to its underlying state variables.
+ * For the most part, this class delegates thread-safety to its underlying
+ * state variables. However, the {@link #trackDequedCommitOf(Thread)} and
+ * {@link #trackExecutedCommitOf(Thread)} method must be called in
+ * sequence within the same transaction, because they are linked
+ * via the {@link #currentCommit} field.
*/
class CommitsTracker {
private final String[] threadGroups;
@@ -59,46 +65,54 @@ class CommitsTracker {
static final class Commit {
private final String threadName;
private final WeakReference thread;
+private final Supplier gcGeneration;
private long queued;
private long dequeued;
private long applied;
-Commit(Thread thread) {
+Commit(Thread thread, Supplier gcGeneration) {
this.threadName = thread.getName();
+this.gcGeneration = gcGeneration;
this.thread = new WeakReference<>(thread);
}
+@NotNull
Commit queued() {
queued = System.currentTimeMillis();
return this;
}
+@NotNull
Commit dequeued() {
dequeued = System.currentTimeMillis();
return this;
}
+@NotNull
Commit applied() {
applied = System.currentTimeMillis();
return this;
}
-String getStackTrace() {
+@Nullable
+StackTraceElement[] getStackTrace() {
Thread t = thread.get();
-if (t != null) {
-StringBuilder threadDetails = new StringBuilder();
-Stream.of(t.getStackTrace()).forEach(threadDetails::append);
-return threadDetails.toString();
-} else {
-return "N/A";
-}
+return t == null
+? null
+: t.getStackTrace();
}
+@NotNull
String getThreadName() {
return threadName;
}
+@Nullable
+GCGeneration getGCGeneration() {
+return gcGeneration.get();
+}
+
long getQueued() {
return queued;
}
@@ -118,8 +132,8 @@ class CommitsTracker {
this.queuedWritersMap = new ConcurrentHashMap<>();
}
-public void trackQueuedCommitOf(Thread thread) {
-queuedWritersMap.put(thread.getName(), new Commit(thread).queued());
+public void trackQueuedCommitOf(Thread thread, Supplier
gcGeneration) {
+queuedWritersMap.put(thread.getName(), new Commit(thread,
gcGeneration).queued());
}
public void trackDequedCommitOf(Thread thread) {
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreMonitor.java
URL:
http://svn.apache.org/viewv
svn commit: r1855586 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
Author: mduerig
Date: Fri Mar 15 13:34:45 2019
New Revision: 1855586
URL: http://svn.apache.org/viewvc?rev=1855586=rev
Log:
OAK-8094: JMX monitoring to detect commits carrying over from previous GC
generation can block other threads from committing
Always handle commits as queued even when the lock is free to avoid races
between checking the number of available permits and actually acquiring the lock
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java?rev=1855586=1855585=1855586=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
Fri Mar 15 13:34:45 2019
@@ -253,26 +253,17 @@ public class LockBasedScheduler implemen
@Override
public NodeState schedule(@NotNull Commit commit, SchedulerOption...
schedulingOptions)
throws CommitFailedException {
-boolean queued = false;
-
try {
commitSemaphoreLogging.warnOnBlockingCommit();
-long queuedTime = -1;
-
-if (commitSemaphore.availablePermits() < 1) {
-queuedTime = System.nanoTime();
-stats.onCommitQueued(Thread.currentThread());
-queued = true;
-}
+long queuedTime = System.nanoTime();
+stats.onCommitQueued(Thread.currentThread());
commitSemaphore.acquire();
commitSemaphoreLogging.commitStarted(commit);
try {
-if (queued) {
-long dequeuedTime = System.nanoTime();
-stats.onCommitDequeued(Thread.currentThread(),
dequeuedTime - queuedTime);
-}
+long dequeuedTime = System.nanoTime();
+stats.onCommitDequeued(Thread.currentThread(), dequeuedTime -
queuedTime);
long beforeCommitTime = System.nanoTime();
svn commit: r1855588 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ test/java/org/apache/jackrabbit/oak/segment/
Author: mduerig
Date: Fri Mar 15 13:34:56 2019
New Revision: 1855588
URL: http://svn.apache.org/viewvc?rev=1855588=rev
Log:
OAK-8094: JMX monitoring to detect commits carrying over from previous GC
generation can block other threads from committing
Add timestamp to monitoring of queued commits
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStats.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStatsMBean.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java?rev=1855588=1855587=1855588=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Fri Mar 15 13:34:56 2019
@@ -22,6 +22,7 @@ package org.apache.jackrabbit.oak.segmen
import static com.google.common.collect.Maps.newHashMap;
import static com.google.common.collect.Queues.newConcurrentLinkedQueue;
+import java.lang.ref.WeakReference;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
@@ -46,36 +47,48 @@ import com.googlecode.concurrentlinkedha
class CommitsTracker {
private final String[] threadGroups;
private final int otherWritersLimit;
-private final boolean collectStackTraces;
-private final ConcurrentMap queuedWritersMap;
+private final ConcurrentMap queuedWritersMap;
private final Queue commits = newConcurrentLinkedQueue();
-private static final class Commit {
-final long timeStamp;
-final String thread;
+static final class Commit {
+private final long timeStamp;
+private final String threadName;
+private final WeakReference thread;
-Commit(long timeStamp, String thread) {
+Commit(long timeStamp, Thread thread) {
this.timeStamp = timeStamp;
-this.thread = thread;
+this.threadName = thread.getName();
+this.thread = new WeakReference<>(thread);
+}
+
+String getStackTrace() {
+Thread t = thread.get();
+if (t != null) {
+StringBuilder threadDetails = new StringBuilder();
+Stream.of(t.getStackTrace()).forEach(threadDetails::append);
+return threadDetails.toString();
+} else {
+return "N/A";
+}
+}
+
+String getThreadName() {
+return threadName;
+}
+
+long getTimeStamp() {
+return timeStamp;
}
}
-CommitsTracker(String[] threadGroups, int otherWritersLimit, boolean
collectStackTraces) {
+CommitsTracker(String[] threadGroups, int otherWritersLimit) {
this.threadGroups = threadGroups;
this.otherWritersLimit = otherWritersLimit;
-this.collectStackTraces = collectStackTraces;
this.queuedWritersMap = new ConcurrentHashMap<>();
}
public void trackQueuedCommitOf(Thread t) {
-String writerDetails = "N/A";
-if (collectStackTraces) {
-StringBuilder threadDetails = new StringBuilder();
-Stream.of(t.getStackTrace()).forEach(threadDetails::append);
-writerDetails = threadDetails.toString();
-}
-
-queuedWritersMap.put(t.getName(), writerDetails);
+queuedWritersMap.put(t.getName(), new
Commit(System.currentTimeMillis(), t));
}
public void trackDequedCommitOf(Thread t) {
@@ -95,10 +108,10 @@ class CommitsTracker {
break;
}
}
-commits.offer(new Commit(t, thread.getName()));
+commits.offer(new Commit(t, thread));
}
-public Map getQueuedWritersMap() {
+public Map getQueuedWritersMap() {
return new HashMap<>(queuedWritersMap);
}
@@ -121,7 +134,7 @@ class CommitsTracker {
long t = System.currentTimeMillis() - 6;
for (Commit commit : commits) {
if (commit.timeStamp > t) {
-String group = findGroupFor(commit.thread);
+String group = findGroupFor(commit.threadName);
if (!"other".equals(group)) {
commitsPerGroup.compute(group, (w, v) -> v == null ? 1 : v
+ 1);
}
@@ -136,9 +149,9 @@ class Commi
svn commit: r1855589 - in /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment: CommitsTracker.java SegmentNodeStoreStats.java SegmentNodeStoreStatsMBean.java
Author: mduerig
Date: Fri Mar 15 13:35:01 2019
New Revision: 1855589
URL: http://svn.apache.org/viewvc?rev=1855589=rev
Log:
OAK-8094: JMX monitoring to detect commits carrying over from previous GC
generation can block other threads from committing
Track and expose commit in progress and queued, dequed and applied time stamps
per commit
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStats.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStatsMBean.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java?rev=1855589=1855588=1855589=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Fri Mar 15 13:35:01 2019
@@ -32,6 +32,7 @@ import java.util.concurrent.ConcurrentMa
import java.util.stream.Stream;
import com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
+import org.jetbrains.annotations.Nullable;
/**
* A simple tracker for the source of commits (writes) in
@@ -50,17 +51,39 @@ class CommitsTracker {
private final ConcurrentMap queuedWritersMap;
private final Queue commits = newConcurrentLinkedQueue();
+/*
+ * Read access via getCurrentWriter() happens usually on a separate
thread, thus volatile
+ */
+private volatile Commit currentCommit;
+
static final class Commit {
-private final long timeStamp;
private final String threadName;
private final WeakReference thread;
-Commit(long timeStamp, Thread thread) {
-this.timeStamp = timeStamp;
+private long queued;
+private long dequeued;
+private long applied;
+
+Commit(Thread thread) {
this.threadName = thread.getName();
this.thread = new WeakReference<>(thread);
}
+Commit queued() {
+queued = System.currentTimeMillis();
+return this;
+}
+
+Commit dequeued() {
+dequeued = System.currentTimeMillis();
+return this;
+}
+
+Commit applied() {
+applied = System.currentTimeMillis();
+return this;
+}
+
String getStackTrace() {
Thread t = thread.get();
if (t != null) {
@@ -76,8 +99,16 @@ class CommitsTracker {
return threadName;
}
-long getTimeStamp() {
-return timeStamp;
+long getQueued() {
+return queued;
+}
+
+long getDequeued() {
+return dequeued;
+}
+
+long getApplied() {
+return applied;
}
}
@@ -87,12 +118,15 @@ class CommitsTracker {
this.queuedWritersMap = new ConcurrentHashMap<>();
}
-public void trackQueuedCommitOf(Thread t) {
-queuedWritersMap.put(t.getName(), new
Commit(System.currentTimeMillis(), t));
+public void trackQueuedCommitOf(Thread thread) {
+queuedWritersMap.put(thread.getName(), new Commit(thread).queued());
}
-public void trackDequedCommitOf(Thread t) {
-queuedWritersMap.remove(t.getName());
+public void trackDequedCommitOf(Thread thread) {
+currentCommit = queuedWritersMap.remove(thread.getName());
+if (currentCommit != null) {
+currentCommit.dequeued();
+}
}
public void trackExecutedCommitOf(Thread thread) {
@@ -102,19 +136,29 @@ class CommitsTracker {
// Purge the queue
// Avoiding removeIf allows us to bail out early. See OAK-7885
while (it.hasNext()) {
-if (it.next().timeStamp < t - 6) {
+if (it.next().getQueued() < t - 6) {
it.remove();
} else {
break;
}
}
-commits.offer(new Commit(t, thread));
+
+if (currentCommit != null) {
+currentCommit.applied();
+commits.offer(currentCommit);
+currentCommit = null;
+}
}
public Map getQueuedWritersMap() {
return new HashMap<>(queuedWritersMap);
}
+@Nullable
+public Commit getCurrentWriter() {
+return currentCommit;
+}
+
private String findGroupFor(String thread) {
if (threadGroups == null) {
return "other";
@@ -133,10 +177,10 @@ class CommitsTrac
svn commit: r1855587 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
Author: mduerig
Date: Fri Mar 15 13:34:49 2019
New Revision: 1855587
URL: http://svn.apache.org/viewvc?rev=1855587=rev
Log:
OAK-8094: JMX monitoring to detect commits carrying over from previous GC
generation can block other threads from committing
Simplify CommitsTrackerTest
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java?rev=1855587=1855586=1855587=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
Fri Mar 15 13:34:49 2019
@@ -19,84 +19,74 @@
package org.apache.jackrabbit.oak.segment;
-import static java.util.concurrent.Executors.newFixedThreadPool;
+import static com.google.common.collect.Lists.newArrayList;
+import static java.lang.Math.min;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
+import java.util.List;
import java.util.Map;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.jackrabbit.oak.commons.concurrent.ExecutorCloser;
import org.junit.Test;
public class CommitsTrackerTest {
-static class DequedCommitTask implements Runnable {
+private static class CommitTask {
private final CommitsTracker commitsTracker;
-private final String threadName;
-private final CountDownLatch latch;
+private final Thread thread;
-public DequedCommitTask(CommitsTracker commitsTracker, String
threadName, CountDownLatch latch) {
+CommitTask(CommitsTracker commitsTracker) {
this.commitsTracker = commitsTracker;
-this.threadName = threadName;
-this.latch = latch;
+this.thread = new Thread();
}
-@Override
-public void run() {
-Thread.currentThread().setName(threadName);
-commitsTracker.trackDequedCommitOf(Thread.currentThread());
-latch.countDown();
+CommitTask(CommitsTracker commitsTracker, String threadName) {
+this.commitsTracker = commitsTracker;
+this.thread = new Thread(threadName);
+}
+
+public void queued() {
+commitsTracker.trackQueuedCommitOf(thread);
+}
+
+public void dequeue() {
+commitsTracker.trackDequedCommitOf(thread);
+}
+
+public void executed() {
+commitsTracker.trackExecutedCommitOf(thread);
+}
+
+public String getThreadName() {
+return thread.getName();
}
}
@Test
public void testCommitsCountOthers() throws InterruptedException {
-CommitsTracker commitsTracker = new CommitsTracker(new String[] {},
10, false);
-ExecutorService executorService = newFixedThreadPool(30);
-final CountDownLatch addLatch = new CountDownLatch(25);
-
-Runnable executedCommitTask = () -> {
-commitsTracker.trackExecutedCommitOf(Thread.currentThread());
-addLatch.countDown();
-};
-
-Runnable queuedCommitTask = () -> {
-Thread t = Thread.currentThread();
-commitsTracker.trackQueuedCommitOf(t);
-addLatch.countDown();
-};
-
-try {
-for (int i = 0; i < 20; i++) {
-executorService.submit(executedCommitTask);
-}
-
-for (int i = 0; i < 5; i++) {
-executorService.submit(queuedCommitTask);
-}
-
-addLatch.await();
-Map commitsCountOthersMap =
commitsTracker.getCommitsCountOthers();
-Map queuedWritersMap =
commitsTracker.getQueuedWritersMap();
-
-assertTrue(commitsCountOthersMap.size() >= 10);
-assertTrue(commitsCountOthersMap.size() < 20);
-assertEquals(5, queuedWritersMap.size());
-
-CountDownLatch removeLatch = new CountDownLatch(5);
-for (String threadName : queuedWritersMap.keySet()) {
-executorService.submit(new DequedCommitTask(commitsTracker,
threadName, removeLatch));
-}
-
-removeLatch.await();
-queuedWritersMap = commitsTracker.getQueuedWritersMap();
-assertEquals(0, queuedWritersMap.size());
-} finally {
-new ExecutorCloser(executorService).close();
+CommitsTracker co
svn commit: r1854596 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Author: mduerig
Date: Fri Mar 1 14:46:35 2019
New Revision: 1854596
URL: http://svn.apache.org/viewvc?rev=1854596=rev
Log:
OAK-8066: Nodes with many direct children can lead to OOME when saving
Flush modified child nodes to disk in regular intervals
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java?rev=1854596=1854595=1854596=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Fri Mar 1 14:46:35 2019
@@ -33,7 +33,6 @@ import static com.google.common.collect.
import static com.google.common.collect.Lists.partition;
import static com.google.common.collect.Maps.newHashMap;
import static com.google.common.io.ByteStreams.read;
-import static java.lang.Integer.getInteger;
import static java.lang.Long.numberOfLeadingZeros;
import static java.lang.Math.min;
import static java.util.Arrays.asList;
@@ -44,6 +43,7 @@ import static org.apache.jackrabbit.oak.
import static org.apache.jackrabbit.oak.api.Type.NAME;
import static org.apache.jackrabbit.oak.api.Type.NAMES;
import static org.apache.jackrabbit.oak.api.Type.STRING;
+import static
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState;
import static org.apache.jackrabbit.oak.segment.MapEntry.newModifiedMapEntry;
import static org.apache.jackrabbit.oak.segment.MapRecord.BUCKETS_PER_LEVEL;
import static
org.apache.jackrabbit.oak.segment.RecordWriters.newNodeStateWriter;
@@ -70,7 +70,6 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.apache.jackrabbit.oak.segment.spi.persistence.Buffer;
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
-import org.apache.jackrabbit.oak.spi.state.ChildNodeEntry;
import org.apache.jackrabbit.oak.spi.state.DefaultNodeStateDiff;
import org.apache.jackrabbit.oak.spi.state.NodeState;
import org.jetbrains.annotations.NotNull;
@@ -89,17 +88,10 @@ public class DefaultSegmentWriter implem
private static final Logger LOG =
LoggerFactory.getLogger(DefaultSegmentWriter.class);
/**
- * Default threshold of the number of modified child nodes of a node after
which to
- * log a warning.
+ * Number of updates to child nodes before changes are flushed to disk.
*/
-public static final int CHILD_NODE_COUNT_WARN_THRESHOLD_DEFAULT = 100;
-
-/**
- * Threshold of the number of modified child nodes of a node after which
to log
- * a warning.
- */
-private static final int CHILD_NODE_COUNT_WARN_THRESHOLD =
-getInteger("oak.segment.childNodeCountWarnThreshold",
CHILD_NODE_COUNT_WARN_THRESHOLD_DEFAULT);
+private static final int CHILD_NODE_UPDATE_LIMIT = Integer
+.getInteger("child.node.update.limit", 1);
@NotNull
private final WriterCacheManager cacheManager;
@@ -201,7 +193,7 @@ public class DefaultSegmentWriter implem
@NotNull
public RecordId writeNode(@NotNull final NodeState state, @Nullable final
Buffer stableIdBytes) throws IOException {
return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
-.writeNode("/", state, stableIdBytes);
+.writeNode(state, stableIdBytes);
}
/**
@@ -219,8 +211,6 @@ public class DefaultSegmentWriter implem
private final Cache nodeCache;
-private int childNodeCountWarnThreshold =
CHILD_NODE_COUNT_WARN_THRESHOLD;
-
SegmentWriteOperation(@NotNull GCGeneration gcGeneration) {
int generation = gcGeneration.getGeneration();
this.gcGeneration = gcGeneration;
@@ -756,10 +746,7 @@ public class DefaultSegmentWriter implem
return tid;
}
-private RecordId writeNode(
-@NotNull String path,
-@NotNull NodeState state,
-@Nullable Buffer stableIdBytes)
+private RecordId writeNode(@NotNull NodeState state, @Nullable Buffer
stableIdBytes)
throws IOException {
RecordId compactedId = deduplicateNode(state);
@@ -770,7 +757,7 @@ public class DefaultSegmentWriter implem
if (state instanceof SegmentNodeState && stableIdBytes == null) {
stableIdBytes = ((SegmentNodeState) state).getStableIdBytes();
}
-RecordId recordId = writeNodeUncached(path, state, stableIdBytes);
+
svn commit: r1854580 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Author: mduerig
Date: Fri Mar 1 12:51:34 2019
New Revision: 1854580
URL: http://svn.apache.org/viewvc?rev=1854580=rev
Log:
OAK-8066: Nodes with many direct children can lead to OOME when saving
Removed left over comment
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java?rev=1854580=1854579=1854580=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Fri Mar 1 12:51:34 2019
@@ -37,9 +37,6 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
-// *The test is disabled by default, to run it you need to set the
{@code SegmentCompactionIT} system property:
-// * {@code mvn test -Dtest=SegmentCompactionIT -Dtest.opts.memory=-Xmx4G}
-
/**
* This test asserts that a large number of child nodes can be added in a
single
* transaction. Due to its long running time the test needs to be explicitly
enabled
svn commit: r1854579 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Author: mduerig
Date: Fri Mar 1 12:49:12 2019
New Revision: 1854579
URL: http://svn.apache.org/viewvc?rev=1854579=rev
Log:
OAK-8066: Nodes with many direct children can lead to OOME when saving
Test case
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java?rev=1854579=auto
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
(added)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/LargeNumberOfChildNodeUpdatesIT.java
Fri Mar 1 12:49:12 2019
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.jackrabbit.oak.segment.file;
+
+import static java.lang.System.getProperty;
+import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+
+import org.apache.jackrabbit.oak.segment.DefaultSegmentWriter;
+import org.apache.jackrabbit.oak.segment.RecordId;
+import org.apache.jackrabbit.oak.segment.SegmentNodeBuilder;
+import org.apache.jackrabbit.oak.segment.SegmentNodeState;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+// *The test is disabled by default, to run it you need to set the
{@code SegmentCompactionIT} system property:
+// * {@code mvn test -Dtest=SegmentCompactionIT -Dtest.opts.memory=-Xmx4G}
+
+/**
+ * This test asserts that a large number of child nodes can be added in a
single
+ * transaction. Due to its long running time the test needs to be explicitly
enabled
+ * via {@code -Dtest=LargeNumberOfChildNodeUpdatesIT}.
+ * Used {@code -DLargeNumberOfChildNodeUpdatesIT.child-count=} to control
the number
+ * of child nodes used by this test. Default is 500.
+ */
+public class LargeNumberOfChildNodeUpdatesIT {
+
+/** Only run if explicitly asked to via
-Dtest=LargeNumberOfChildNodeUpdatesIT */
+private static final boolean ENABLED =
+
LargeNumberOfChildNodeUpdatesIT.class.getSimpleName().equals(getProperty("test"));
+
+private static final int NODE_COUNT = Integer
+.getInteger("LargeNumberOfChildNodeUpdatesIT.child-count",
500);
+
+@Rule
+public final TemporaryFolder folder = new TemporaryFolder(new
File("target"));
+
+@Before
+public void setup() throws IOException, InvalidFileStoreVersionException {
+assumeTrue(ENABLED);
+}
+
+@Test
+public void testNode() throws IOException,
InvalidFileStoreVersionException {
+try (FileStore fileStore =
FileStoreBuilder.fileStoreBuilder(folder.getRoot()).build()) {
+DefaultSegmentWriter writer = defaultSegmentWriterBuilder("test")
+.withGeneration(GCGeneration.newGCGeneration(1, 1, false))
+.build(fileStore);
+
+SegmentNodeState root = fileStore.getHead();
+SegmentNodeBuilder builder = root.builder();
+for (int k = 0; k < NODE_COUNT; k++) {
+builder.setChildNode("n-" + k);
+}
+
+SegmentNodeState node1 = builder.getNodeState();
+RecordId nodeId = writer.writeNode(node1);
+SegmentNodeState node2 = fileStore.getReader().readNode(nodeId);
+
+assertNotEquals(node1.getRecordId(), node2.getRecordId());
+assertEquals(node1, node2);
+}
+}
+
+}
svn commit: r1854559 - in /jackrabbit/oak/branches/1.10: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/
Author: mduerig
Date: Fri Mar 1 07:20:15 2019
New Revision: 1854559
URL: http://svn.apache.org/viewvc?rev=1854559=rev
Log:
OAK-8071: Logging to detect commits carrying over from previous GC generation
can block other threads from committing
Merged r1854515
Modified:
jackrabbit/oak/branches/1.10/ (props changed)
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
Propchange: jackrabbit/oak/branches/1.10/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Fri Mar 1 07:20:15 2019
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1850874,1850882,1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853229,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044,1854058,1854113,1854373,1854377,1854380,1854385,1854401,1854403,1854462,1854466,1854468
+/jackrabbit/oak/trunk:1850874,1850882,1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853229,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044,1854058,1854113,1854373,1854377,1854380,1854385,1854401,1854403,1854462,1854466,1854468,1854515
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java?rev=1854559=1854558=1854559=diff
==
---
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
(original)
+++
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
Fri Mar 1 07:20:15 2019
@@ -18,6 +18,7 @@
*/
package org.apache.jackrabbit.oak.segment;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.jetbrains.annotations.NotNull;
/**
@@ -83,6 +84,16 @@ class Record {
return new RecordId(segmentId, recordNumber);
}
+/**
+ * Get the underlying segment's gc generation. Might cause the segment to
+ * get loaded if the generation info is missing
+ * @return the segment's gc generation
+ */
+@NotNull
+public GCGeneration getGcGeneration() {
+return segmentId.getGcGeneration();
+}
+
//< Object >--
@Override
Modified:
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java?rev=1854559=1854558=1854559=diff
==
---
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
(original)
+++
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
Fri Mar 1 07:20:15 2019
@@ -23,12 +23,14 @@ import static com.google.common.base.Pre
import org.apache.jackrabbit.oak.api.CommitFailedException;
import org.apache.jackrabbit.oak.segment.SegmentNodeBuilder;
import org.apache.jackrabbit.oak.segment.SegmentNodeState;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.apache.jackrabbit.oak.spi.commit.CommitHook;
import org.apache.jackrabbit.oak.spi.commit.CommitInfo;
import org.apache.jackrabbit.oak.spi.state.ConflictAnnotatingRebaseDiff;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeState;
import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
/**
* A {@code Commit} instance represents a set of related changes, which when
@@ -41,6 +43,8 @@ public class Commit {
private final CommitHook hook;
private final CommitInfo info;
+private volatile GCGeneration gcGeneration;
+
public Commit(@NotNull NodeBuilder changes, @NotNull CommitHook hook,
@NotNull CommitInfo info) {
checkNotNull(changes);
checkArgument(changes instanceof SegmentNodeBuilder);
@@ -51,6 +55,30 @@ public class Commit {
}
/**
+ * This method makes a best effort on getting the gc generation of the
current commit.
+ * However it avoids causing a write
svn commit: r1854560 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/
Author: mduerig Date: Fri Mar 1 07:20:30 2019 New Revision: 1854560 URL: http://svn.apache.org/viewvc?rev=1854560=rev Log: OAK-8071: Logging to detect commits carrying over from previous GC generation can block other threads from committing > Merged r1854515 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Mar 1 07:20:30 2019 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822182,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828827,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347 ,1830748,1830911,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843621,1843652,1843911,1844070,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846396,1846429,1846581,1846617,1847096,1848073,1848181-1848182,1848191,1848217,1848822-1848823,1850221,1850837,1850 874,1851533-1851535,1851619,1852120,1852451,1852492,1852528,1852582,1852584,1853141,1853229,1853393,1853429,1853433,1853866,1853870,1853893,1853969,1853997,1854044,1854466 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822182,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828827,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347 ,1830748,1830911,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843621,1843652,1843911,1844070,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845
svn commit: r1854515 - in /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment: Record.java scheduler/Commit.java scheduler/LockBasedScheduler.java
Author: mduerig
Date: Thu Feb 28 09:58:13 2019
New Revision: 1854515
URL: http://svn.apache.org/viewvc?rev=1854515=rev
Log:
OAK-8071: Logging to detect commits carrying over from previous GC generation
can block other threads from committing
Log warnings for commits being blocked on the current commit taking a long time
or the current commit crossing a gc boundary.
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java?rev=1854515=1854514=1854515=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Record.java
Thu Feb 28 09:58:13 2019
@@ -18,6 +18,7 @@
*/
package org.apache.jackrabbit.oak.segment;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.jetbrains.annotations.NotNull;
/**
@@ -83,6 +84,16 @@ class Record {
return new RecordId(segmentId, recordNumber);
}
+/**
+ * Get the underlying segment's gc generation. Might cause the segment to
+ * get loaded if the generation info is missing
+ * @return the segment's gc generation
+ */
+@NotNull
+public GCGeneration getGcGeneration() {
+return segmentId.getGcGeneration();
+}
+
//< Object >--
@Override
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java?rev=1854515=1854514=1854515=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/Commit.java
Thu Feb 28 09:58:13 2019
@@ -23,12 +23,14 @@ import static com.google.common.base.Pre
import org.apache.jackrabbit.oak.api.CommitFailedException;
import org.apache.jackrabbit.oak.segment.SegmentNodeBuilder;
import org.apache.jackrabbit.oak.segment.SegmentNodeState;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.apache.jackrabbit.oak.spi.commit.CommitHook;
import org.apache.jackrabbit.oak.spi.commit.CommitInfo;
import org.apache.jackrabbit.oak.spi.state.ConflictAnnotatingRebaseDiff;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeState;
import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
/**
* A {@code Commit} instance represents a set of related changes, which when
@@ -41,6 +43,8 @@ public class Commit {
private final CommitHook hook;
private final CommitInfo info;
+private volatile GCGeneration gcGeneration;
+
public Commit(@NotNull NodeBuilder changes, @NotNull CommitHook hook,
@NotNull CommitInfo info) {
checkNotNull(changes);
checkArgument(changes instanceof SegmentNodeBuilder);
@@ -51,6 +55,30 @@ public class Commit {
}
/**
+ * This method makes a best effort on getting the gc generation of the
current commit.
+ * However it avoids causing a write ahead action by calling {@link
NodeBuilder#getName(String)}
+ * on the changes in this commit.
+ *
+ * @return the gc generation of this commit or {@code null} if not yet
available.
+ */
+@Nullable
+public GCGeneration getGCGeneration() {
+return gcGeneration;
+}
+
+@NotNull
+private NodeState getBeforeState() {
+return changes.getBaseState();
+}
+
+@NotNull
+private SegmentNodeState getAfterState() {
+SegmentNodeState after = changes.getNodeState();
+gcGeneration = after.getGcGeneration();
+return after;
+}
+
+/**
* Apply the changes represented by this commit to the passed {@code base}
* node state.
*
@@ -63,19 +91,19 @@ public class Commit {
*/
public SegmentNodeState apply(SegmentNodeState base) throws
CommitFailedException {
SegmentNodeBuilder builder = base.builder();
-if (SegmentNodeState.fastEquals(changes.getBaseState(),
base.getChildNod
svn commit: r1854129 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Author: mduerig Date: Fri Feb 22 10:02:51 2019 New Revision: 1854129 URL: http://svn.apache.org/viewvc?rev=1854129=rev Log: OAK-8069: Log warning for too many transient modifications of direct child nodes Merged r1854058 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Feb 22 10:02:51 2019 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846396,1846429,1846617,1848073,1848181-1848182,1848191,1848217,1848822-1848823,1850221,1850837,1851533-1851535,1851619,1852120,1852451,1852492,1852 528,1852582,1852584,1853393,1853429,1853433,1853866,1853870,1853893,1853969,1853997 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846396,1846429,1846617,1848073,1848181-1848182,1848191,1848217,1848822-1848823,1850221,1850837,1851533-1851535,1851619,1852120,1852451,1852492,1852 528,1852582,1852584,1853393,1853429,1853433,1853866,1853870,1853893,1853969,1853997,1854058 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java URL
svn commit: r1854125 - in /jackrabbit/oak/branches/1.10: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Author: mduerig
Date: Fri Feb 22 09:49:33 2019
New Revision: 1854125
URL: http://svn.apache.org/viewvc?rev=1854125=rev
Log:
OAK-8069: Log warning for too many transient modifications of direct child nodes
Merged r1854058
Modified:
jackrabbit/oak/branches/1.10/ (props changed)
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Propchange: jackrabbit/oak/branches/1.10/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Fri Feb 22 09:49:33 2019
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044
+/jackrabbit/oak/trunk:1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044,1854058
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java?rev=1854125=1854124=1854125=diff
==
---
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
(original)
+++
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Fri Feb 22 09:49:33 2019
@@ -33,6 +33,7 @@ import static com.google.common.collect.
import static com.google.common.collect.Lists.partition;
import static com.google.common.collect.Maps.newHashMap;
import static com.google.common.io.ByteStreams.read;
+import static java.lang.Integer.getInteger;
import static java.lang.Long.numberOfLeadingZeros;
import static java.lang.Math.min;
import static java.util.Arrays.asList;
@@ -87,6 +88,19 @@ public class DefaultSegmentWriter implem
private static final Logger LOG =
LoggerFactory.getLogger(DefaultSegmentWriter.class);
+/**
+ * Default threshold of the number of modified child nodes of a node after
which to
+ * log a warning.
+ */
+public static final int CHILD_NODE_COUNT_WARN_THRESHOLD_DEFAULT = 100;
+
+/**
+ * Threshold of the number of modified child nodes of a node after which
to log
+ * a warning.
+ */
+private static final int CHILD_NODE_COUNT_WARN_THRESHOLD =
+getInteger("oak.segment.childNodeCountWarnThreshold",
CHILD_NODE_COUNT_WARN_THRESHOLD_DEFAULT);
+
@NotNull
private final WriterCacheManager cacheManager;
@@ -187,7 +201,7 @@ public class DefaultSegmentWriter implem
@NotNull
public RecordId writeNode(@NotNull final NodeState state, @Nullable final
Buffer stableIdBytes) throws IOException {
return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
-.writeNode(state, stableIdBytes);
+.writeNode("/", state, stableIdBytes);
}
/**
@@ -205,6 +219,8 @@ public class DefaultSegmentWriter implem
private final Cache nodeCache;
+private int childNodeCountWarnThreshold =
CHILD_NODE_COUNT_WARN_THRESHOLD;
+
SegmentWriteOperation(@NotNull GCGeneration gcGeneration) {
int generation = gcGeneration.getGeneration();
this.gcGeneration = gcGeneration;
@@ -740,7 +756,10 @@ public class DefaultSegmentWriter implem
return tid;
}
-private RecordId writeNode(@NotNull NodeState state, @Nullable Buffer
stableIdBytes)
+private RecordId writeNode(
+@NotNull String path,
+@NotNull NodeState state,
+@Nullable Buffer stableIdBytes)
throws IOException {
RecordId compactedId = deduplicateNode(state);
@@ -751,7 +770,7 @@ public class DefaultSegmentWriter implem
if (state instanceof SegmentNodeState && stableIdBytes == null) {
stableIdBytes = ((SegmentNodeState) state).getStableIdBytes();
}
-RecordId recordId = writeNodeUncached(state, stableIdBytes);
+RecordId recordId = writeNodeUncached(path, state, stableIdBytes);
if (stableIdBytes != null) {
// This node state has been rewritten because it is from an
older
@@ -767,7 +786,17 @@ public class DefaultSegmentWriter implem
return (byte) (Byte.MIN_VALUE + 64 -
numberOfLeadingZeros(childCount));
}
-private RecordId writeNodeUncached(@NotNull NodeSt
svn commit: r1853814 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java oak-segment-tar/src/test/java/org/apache/jackrab
Author: mduerig Date: Mon Feb 18 16:16:25 2019 New Revision: 1853814 URL: http://svn.apache.org/viewvc?rev=1853814=rev Log: OAK-8033: Node states sometimes refer to more than a single generation of segments after a full compaction Merged revision 1853429 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Mon Feb 18 16:16:25 2019 @@ -1,4 +1,4 @@ /jackrabbit/oak/branches/1.0:1665962 /jackrabbit/oak/branches/1.8:1844835 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814052,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1816019,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1835780,1836082,1837475,1837998,1838637,1839746,1840 024,1840226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845135,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846617,1848181-1848182,1848191,1848217,1848822-1848823,1850837 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814052,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1816019,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648
svn commit: r1853813 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java oak-segment-tar/src/test/java/org/apache/jackrab
Author: mduerig Date: Mon Feb 18 16:16:08 2019 New Revision: 1853813 URL: http://svn.apache.org/viewvc?rev=1853813=rev Log: OAK-8033: Node states sometimes refer to more than a single generation of segments after a full compaction Merged revision 1853429 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Mon Feb 18 16:16:08 2019 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846396,1846429,1846617,1848073,1848181-1848182,1848191,1848217,1848822-1848823,1850837,1851533-1851535,1851619,1852120,1852451,1852492,1853393,1853 433 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1836548,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1840769,1841314,1841352,1842089,1842677,1843175,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846396,1846429,1846617,1848073,1848181-1848182,1848191,1848217,1848822-1848823,1850837,1851533-1851535,1851619,1852120,1852451,1852492,1853393,1853 429,1853433 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java URL
svn commit: r1853812 - in /jackrabbit/oak/branches/1.10: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java oak-segment-tar/src/test/java/org/apache/jackra
Author: mduerig
Date: Mon Feb 18 16:15:35 2019
New Revision: 1853812
URL: http://svn.apache.org/viewvc?rev=1853812=rev
Log:
OAK-8033: Node states sometimes refer to more than a single generation of
segments after a full compaction
Merged revision 1853429
Modified:
jackrabbit/oak/branches/1.10/ (props changed)
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
jackrabbit/oak/branches/1.10/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Propchange: jackrabbit/oak/branches/1.10/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Feb 18 16:15:35 2019
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852920,1853393,1853433
+/jackrabbit/oak/trunk:1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852920,1853393,1853429,1853433
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1853812=1853811=1853812=diff
==
---
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
(original)
+++
jackrabbit/oak/branches/1.10/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Mon Feb 18 16:15:35 2019
@@ -105,7 +105,7 @@ public class SegmentBufferWriterPool imp
@NotNull WriteOperation writeOperation)
throws IOException {
SimpleImmutableEntry key = new
SimpleImmutableEntry<>(currentThread(), gcGeneration);
-SegmentBufferWriter writer = borrowWriter(key);
+SegmentBufferWriter writer = borrowWriter(key, gcGeneration);
try {
return writeOperation.execute(writer);
} finally {
@@ -189,7 +189,7 @@ public class SegmentBufferWriterPool imp
* a fresh writer at any time. Callers need to return a writer before
* borrowing it again. Failing to do so leads to undefined behaviour.
*/
-private SegmentBufferWriter borrowWriter(Object key) {
+private SegmentBufferWriter borrowWriter(@NotNull Object key, @NotNull
GCGeneration gcGeneration) {
poolMonitor.enter();
try {
SegmentBufferWriter writer = writers.remove(key);
@@ -198,7 +198,7 @@ public class SegmentBufferWriterPool imp
idProvider,
reader,
getWriterId(wid),
-gcGeneration.get()
+gcGeneration
);
}
borrowed.add(writer);
Modified:
jackrabbit/oak/branches/1.10/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java?rev=1853812=1853811=1853812=diff
==
---
jackrabbit/oak/branches/1.10/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
(original)
+++
jackrabbit/oak/branches/1.10/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Mon Feb 18 16:15:35 2019
@@ -21,6 +21,7 @@ package org.apache.jackrabbit.oak.segmen
import static com.google.common.collect.Lists.newArrayList;
import static com.google.common.collect.Sets.newHashSet;
+import static
com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly;
import static
com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static java.lang.Integer.getInteger;
import static java.lang.String.valueOf;
@@ -51,6 +52,7 @@ import java.util.Random;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
@@ -59,6 +61,7 @@ import java.util.concurrent.ScheduledExe
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
import com.google.common.io.ByteStreams;
import org.apache.jackrabbit.oak.api.Blob;
@@ -68,6 +71,7 @@ import org.apache.jackrabbit.oak.api.Typ
import org.apache.jackrabbit.oak.commons.concurrent.ExecutorCloser;
svn commit: r1853429 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java test/java/org/apache/jackrabbit/oak/segment/CompactionAndC
Author: mduerig
Date: Tue Feb 12 12:29:19 2019
New Revision: 1853429
URL: http://svn.apache.org/viewvc?rev=1853429=rev
Log:
OAK-8033: Node states sometimes refer to more than a single generation of
segments after a full compaction
Use the gc-generation of the current write operation instead of the one of the
repository head state when allocating new segment buffer writers in the course
of a write operation.
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1853429=1853428=1853429=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Tue Feb 12 12:29:19 2019
@@ -184,25 +184,21 @@ public class SegmentBufferWriterPool imp
}
}
-
/**
* Return a writer from the pool by its {@code key}. This method may return
* a fresh writer at any time. Callers need to return a writer before
* borrowing it again. Failing to do so leads to undefined behaviour.
*/
-private SegmentBufferWriter borrowWriter(Object key, GCGeneration
gcGeneration) {
+private SegmentBufferWriter borrowWriter(@NotNull Object key, @NotNull
GCGeneration gcGeneration) {
poolMonitor.enter();
try {
SegmentBufferWriter writer = writers.remove(key);
if (writer == null) {
-GCGeneration thisGeneration = this.gcGeneration.get();
-checkState(thisGeneration.equals(gcGeneration),
-"Mismatching GC generations " + thisGeneration + " != " +
gcGeneration);
writer = new SegmentBufferWriter(
idProvider,
reader,
getWriterId(wid),
-thisGeneration
+gcGeneration
);
}
borrowed.add(writer);
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java?rev=1853429=1853428=1853429=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Tue Feb 12 12:29:19 2019
@@ -21,6 +21,7 @@ package org.apache.jackrabbit.oak.segmen
import static com.google.common.collect.Lists.newArrayList;
import static com.google.common.collect.Sets.newHashSet;
+import static
com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly;
import static
com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static java.lang.Integer.getInteger;
import static java.lang.String.valueOf;
@@ -51,6 +52,7 @@ import java.util.Random;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
@@ -59,6 +61,7 @@ import java.util.concurrent.ScheduledExe
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
import com.google.common.io.ByteStreams;
import org.apache.jackrabbit.oak.api.Blob;
@@ -68,6 +71,7 @@ import org.apache.jackrabbit.oak.api.Typ
import org.apache.jackrabbit.oak.commons.concurrent.ExecutorCloser;
import org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore;
import org.apache.jackrabbit.oak.plugins.blob.datastore.OakFileDataStore;
+import org.apache.jackrabbit.oak.plugins.memory.StringPropertyState;
import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions;
import org.apache.jackrabbit.oak.segment.file.FileStore;
import org.apache.jackrabbit.oak.segment.file.FileStoreGCMonitor;
@@ -648,37 +652,36 @@ public class CompactionAndCleanupIT {
.withMaxFileSize(2)
.withMemoryMapping(true)
svn commit: r1853389 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Author: mduerig
Date: Mon Feb 11 16:47:17 2019
New Revision: 1853389
URL: http://svn.apache.org/viewvc?rev=1853389=rev
Log:
OAK-8039: Intermittent test failure of CompactionAndCleanupIT.testMixedSegments
Better error message in assertion
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1853389=1853388=1853389=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Mon Feb 11 16:47:17 2019
@@ -196,7 +196,8 @@ public class SegmentBufferWriterPool imp
SegmentBufferWriter writer = writers.remove(key);
if (writer == null) {
GCGeneration thisGeneration = this.gcGeneration.get();
-checkState(thisGeneration.equals(gcGeneration));
+checkState(thisGeneration.equals(gcGeneration),
+"Mismatching GC generations " + thisGeneration + " != " +
gcGeneration);
writer = new SegmentBufferWriter(
idProvider,
reader,
svn commit: r1853379 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Author: mduerig
Date: Mon Feb 11 14:18:40 2019
New Revision: 1853379
URL: http://svn.apache.org/viewvc?rev=1853379=rev
Log:
OAK-8039: Intermittent test failure of CompactionAndCleanupIT.testMixedSegments
Include stack traces of exceptions in background thread
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java?rev=1853379=1853378=1853379=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Mon Feb 11 14:18:40 2019
@@ -659,7 +659,7 @@ public class CompactionAndCleanupIT {
collectSegments(store.getReader(), store.getRevisions(),
beforeSegments);
final AtomicReference run = new
AtomicReference(true);
-final List failedCommits = newArrayList();
+final List failedCommits = newArrayList();
Thread[] threads = new Thread[10];
for (int k = 0; k < threads.length; k++) {
final int threadId = k;
@@ -677,7 +677,7 @@ public class CompactionAndCleanupIT {
Thread.interrupted();
break;
} catch (Exception e) {
-failedCommits.add(nodeName);
+failedCommits.add(new ExecutionException("Failed
commit " + nodeName, e));
}
}
}
@@ -692,7 +692,9 @@ public class CompactionAndCleanupIT {
store.flush();
assumeTrue("Failed to acquire compaction lock",
compactionSuccess.get());
-assertTrue("Failed commits: " + failedCommits,
failedCommits.isEmpty());
+for (Exception failedCommit : failedCommits) {
+throw new Exception("A background commit failed", failedCommit);
+}
Set afterSegments = new HashSet();
collectSegments(store.getReader(), store.getRevisions(),
afterSegments);
svn commit: r1853331 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Author: mduerig
Date: Mon Feb 11 08:15:08 2019
New Revision: 1853331
URL: http://svn.apache.org/viewvc?rev=1853331=rev
Log:
OAK-8039: Intermittent test failure of CompactionAndCleanupIT.testMixedSegments
Report all exceptions from background threads
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java?rev=1853331=1853330=1853331=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Mon Feb 11 08:15:08 2019
@@ -673,11 +673,11 @@ public class CompactionAndCleanupIT {
root.setChildNode(nodeName);
nodeStore.merge(root, EmptyHook.INSTANCE,
CommitInfo.EMPTY);
Thread.sleep(5);
-} catch (CommitFailedException e) {
-failedCommits.add(nodeName);
} catch (InterruptedException e) {
Thread.interrupted();
break;
+} catch (Exception e) {
+failedCommits.add(nodeName);
}
}
}
svn commit: r1853332 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Author: mduerig
Date: Mon Feb 11 08:15:12 2019
New Revision: 1853332
URL: http://svn.apache.org/viewvc?rev=1853332=rev
Log:
OAK-8039: Intermittent test failure of CompactionAndCleanupIT.testMixedSegments
Add assertion to root cause of OAK-8014 to determine whether that issue also
causes this issue
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1853332=1853331=1853332=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Mon Feb 11 08:15:12 2019
@@ -105,7 +105,7 @@ public class SegmentBufferWriterPool imp
@NotNull WriteOperation writeOperation)
throws IOException {
SimpleImmutableEntry key = new
SimpleImmutableEntry<>(currentThread(), gcGeneration);
-SegmentBufferWriter writer = borrowWriter(key);
+SegmentBufferWriter writer = borrowWriter(key, gcGeneration);
try {
return writeOperation.execute(writer);
} finally {
@@ -184,21 +184,24 @@ public class SegmentBufferWriterPool imp
}
}
+
/**
* Return a writer from the pool by its {@code key}. This method may return
* a fresh writer at any time. Callers need to return a writer before
* borrowing it again. Failing to do so leads to undefined behaviour.
*/
-private SegmentBufferWriter borrowWriter(Object key) {
+private SegmentBufferWriter borrowWriter(Object key, GCGeneration
gcGeneration) {
poolMonitor.enter();
try {
SegmentBufferWriter writer = writers.remove(key);
if (writer == null) {
+GCGeneration thisGeneration = this.gcGeneration.get();
+checkState(thisGeneration.equals(gcGeneration));
writer = new SegmentBufferWriter(
idProvider,
reader,
getWriterId(wid),
-gcGeneration.get()
+thisGeneration
);
}
borrowed.add(writer);
svn commit: r1848194 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig Date: Wed Dec 5 09:36:48 2018 New Revision: 1848194 URL: http://svn.apache.org/viewvc?rev=1848194=rev Log: OAK-7909: Backport and validate OAK-7867 to Oak 1.6 Merged r1845415 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Dec 5 09:36:48 2018 @@ -1,4 +1,4 @@ /jackrabbit/oak/branches/1.0:1665962 /jackrabbit/oak/branches/1.8:1844835 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832258,1832379,1832535,1833308,1834112,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840 226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845135,1845405,1845730-1845731,1845863,1845865,1846057,1846617 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832258,1832379,1832535,1833308,1834112,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840 226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845135,1845405,1845415,1845730-1845731,1845863,1845865,1846057,1846617 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.6/oak-segment
svn commit: r1848193 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/ oak-segment
Author: mduerig Date: Wed Dec 5 09:34:27 2018 New Revision: 1848193 URL: http://svn.apache.org/viewvc?rev=1848193=rev Log: OAK-7909: Backport and validate OAK-7867 to Oak 1.6 Merged r1845405 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriterBuilder.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriteOperationHandler.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPoolTest.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Dec 5 09:34:27 2018 @@ -1,4 +1,4 @@ /jackrabbit/oak/branches/1.0:1665962 /jackrabbit/oak/branches/1.8:1844835 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832258,1832379,1832535,1833308,1834112,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840 226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845135,1845730-1845731,1845863,1845865,1846057,1846617 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987
svn commit: r1848192 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig Date: Wed Dec 5 09:30:26 2018 New Revision: 1848192 URL: http://svn.apache.org/viewvc?rev=1848192=rev Log: OAK-7909: Backport and validate OAK-7867 to Oak 1.6 Merged r1845135 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Dec 5 09:30:26 2018 @@ -1,4 +1,4 @@ /jackrabbit/oak/branches/1.0:1665962 /jackrabbit/oak/branches/1.8:1844835 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832258,1832379,1832535,1833308,1834112,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840 226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845730-1845731,1845863,1845865,1846057,1846617 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832258,1832379,1832535,1833308,1834112,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840 226,1842677,1843175,1843222,1843231,1844549,1844642,1844728,1845135,1845730-1845731,1845863,1845865,1846057,1846617 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache
svn commit: r1848052 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
Author: mduerig
Date: Mon Dec 3 15:14:25 2018
New Revision: 1848052
URL: http://svn.apache.org/viewvc?rev=1848052=rev
Log:
OAK-7934: Expose the number of segments as metric
OAK-7933: Expose number of tar readers as metric
Move updating the counters outside of the lock in cleanup()
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java?rev=1848052=1848051=1848052=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
Mon Dec 3 15:14:25 2018
@@ -687,11 +687,12 @@ public class TarFiles implements Closeab
Node closeables;
long reclaimed;
+Node swept;
while (true) {
closeables = null;
reclaimed = 0;
-Node swept = null;
+swept = null;
// The following loops creates a modified version of `readers` and
// saves it into `swept`. Some TAR readers in `readers` have been
@@ -754,8 +755,6 @@ public class TarFiles implements Closeab
try {
if (readers == head) {
readers = swept;
-readerCount.dec(getSize(head) - getSize(swept));
-segmentCount.dec(getSegmentCount(head) -
getSegmentCount(swept));
break;
} else {
head = readers;
@@ -764,6 +763,8 @@ public class TarFiles implements Closeab
lock.writeLock().unlock();
}
}
+readerCount.dec(getSize(head) - getSize(swept));
+segmentCount.dec(getSegmentCount(head) - getSegmentCount(swept));
result.reclaimedSize -= reclaimed;
svn commit: r1848050 - in /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file: FileStore.java tar/TarFiles.java
Author: mduerig
Date: Mon Dec 3 15:14:12 2018
New Revision: 1848050
URL: http://svn.apache.org/viewvc?rev=1848050=rev
Log:
OAK-7933: Expose number of tar readers as metric
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1848050=1848049=1848050=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Mon Dec 3 15:14:12 2018
@@ -25,6 +25,7 @@ import static org.apache.jackrabbit.oak.
import static
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.EMPTY_NODE;
import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static
org.apache.jackrabbit.oak.segment.file.PrintableBytes.newPrintableBytes;
+import static org.apache.jackrabbit.oak.stats.StatsOptions.DEFAULT;
import static org.apache.jackrabbit.oak.stats.StatsOptions.METRICS_ONLY;
import java.io.IOException;
@@ -53,6 +54,8 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.spi.persistence.RepositoryLock;
import
org.apache.jackrabbit.oak.segment.spi.persistence.SegmentNodeStorePersistence;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
+import org.apache.jackrabbit.oak.stats.CounterStats;
+import org.apache.jackrabbit.oak.stats.StatisticsProvider;
import org.apache.jackrabbit.oak.stats.TimerStats;
import org.apache.jackrabbit.oak.stats.TimerStats.Context;
import org.jetbrains.annotations.NotNull;
@@ -63,11 +66,15 @@ import org.slf4j.LoggerFactory;
* The storage implementation for tar files.
*/
public class FileStore extends AbstractFileStore {
-
private static final Logger log = LoggerFactory.getLogger(FileStore.class);
-
private static final int MB = 1024 * 1024;
+/**
+ * Name of the {@link CounterStats counter} exposing the number of {@code
TarReader}
+ * instances in use by {@link TarFiles}.
+ */
+private static final String TAR_READER_COUNT = "TAR_READER_COUNT";
+
private static GarbageCollectionStrategy newGarbageCollectionStrategy() {
if (Boolean.getBoolean("gc.classic")) {
return new SynchronizedGarbageCollectionStrategy(new
DefaultGarbageCollectionStrategy());
@@ -125,18 +132,20 @@ public class FileStore extends AbstractF
SegmentNodeStorePersistence persistence = builder.getPersistence();
repositoryLock = persistence.lockRepository();
+StatisticsProvider statsProvider = builder.getStatsProvider();
this.segmentWriter = defaultSegmentWriterBuilder("sys")
.withGeneration(() -> getGcGeneration().nonGC())
.withWriterPool()
.with(builder.getCacheManager()
-.withAccessTracking("WRITE",
builder.getStatsProvider()))
+.withAccessTracking("WRITE", statsProvider))
.build(this);
newManifestChecker(persistence,
builder.getStrictVersionCheck()).checkAndUpdateManifest();
-this.stats = new FileStoreStats(builder.getStatsProvider(), this, 0);
+this.stats = new FileStoreStats(statsProvider, this, 0);
+CounterStats readerCountStats =
statsProvider.getCounterStats(TAR_READER_COUNT, DEFAULT);
TarFiles.Builder tarFilesBuilder = TarFiles.builder()
.withDirectory(directory)
.withMemoryMapping(memoryMapping)
@@ -144,7 +153,8 @@ public class FileStore extends AbstractF
.withIOMonitor(ioMonitor)
.withFileStoreMonitor(stats)
.withMaxFileSize(builder.getMaxFileSize() * MB)
-.withPersistence(builder.getPersistence());
+.withPersistence(builder.getPersistence())
+.withReaderCountStats(readerCountStats);
this.tarFiles = tarFilesBuilder.build();
long size = this.tarFiles.size();
@@ -173,7 +183,7 @@ public class FileStore extends AbstractF
this::flush,
generation ->
defaultSegmentWriterBuilder("c")
-
.with(builder.getCacheManager().withAccessTracking("COMPACT",
builder.getStatsProvider()))
+
.with(builder.getCacheManager().withAccessTracking("COMPACT", statsProvider))
svn commit: r1848051 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/file/ main/java/org/apache/jackrabbit/oak/segment/file/tar/ test/java/org/apache/jackra
Author: mduerig
Date: Mon Dec 3 15:14:20 2018
New Revision: 1848051
URL: http://svn.apache.org/viewvc?rev=1848051=rev
Log:
OAK-7934: Expose the number of segments as metric
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriterTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1848051=1848050=1848051=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Mon Dec 3 15:14:20 2018
@@ -75,6 +75,11 @@ public class FileStore extends AbstractF
*/
private static final String TAR_READER_COUNT = "TAR_READER_COUNT";
+/**
+ * Name of the {@link CounterStats counter} exposing the number of
segments.
+ */
+private static final String SEGMENT_COUNT = "SEGMENT_COUNT";
+
private static GarbageCollectionStrategy newGarbageCollectionStrategy() {
if (Boolean.getBoolean("gc.classic")) {
return new SynchronizedGarbageCollectionStrategy(new
DefaultGarbageCollectionStrategy());
@@ -146,6 +151,7 @@ public class FileStore extends AbstractF
this.stats = new FileStoreStats(statsProvider, this, 0);
CounterStats readerCountStats =
statsProvider.getCounterStats(TAR_READER_COUNT, DEFAULT);
+CounterStats segmentCountStats =
statsProvider.getCounterStats(SEGMENT_COUNT, DEFAULT);
TarFiles.Builder tarFilesBuilder = TarFiles.builder()
.withDirectory(directory)
.withMemoryMapping(memoryMapping)
@@ -154,7 +160,8 @@ public class FileStore extends AbstractF
.withFileStoreMonitor(stats)
.withMaxFileSize(builder.getMaxFileSize() * MB)
.withPersistence(builder.getPersistence())
-.withReaderCountStats(readerCountStats);
+.withReaderCountStats(readerCountStats)
+.withSegmentCountStats(segmentCountStats);
this.tarFiles = tarFilesBuilder.build();
long size = this.tarFiles.size();
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java?rev=1848051=1848050=1848051=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
Mon Dec 3 15:14:20 2018
@@ -130,6 +130,8 @@ public class TarFiles implements Closeab
private CounterStats readerCountStats = NoopStats.INSTANCE;
+private CounterStats segmentCountStats = NoopStats.INSTANCE;
+
private Builder() {
// Prevent external instantiation.
}
@@ -185,6 +187,11 @@ public class TarFiles implements Closeab
return this;
}
+public Builder withSegmentCountStats(CounterStats segmentCountStats) {
+this.segmentCountStats = segmentCountStats;
+return this;
+}
+
public TarFiles build() throws IOException {
checkState(directory != null, "Directory not specified");
checkState(tarRecovery != null, "TAR recovery strategy not
specified");
@@ -343,10 +350,20 @@ public class TarFiles implements Closeab
*/
private final CounterStats readerCount;
+/**
+ * Counter exposing the number of segments.
+ */
+private final CounterStats segmentCount;
+
+private static int getSegmentCount(TarReader reader) {
+return reader.getEntries().length;
+}
+
private TarFiles(Builder builder) throws IOException {
maxFileSize = builder.maxFileSize;
archiveManager = builder.buildArchiveManager();
readerCount = builder.readerCountStats;
+segmentCount = builder.segmentCountStats;
Map> map =
collectFiles(archiveManager);
Integer[] indices = map.keySet().toArray(new Integer[ma
svn commit: r1846988 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig
Date: Tue Nov 20 10:43:49 2018
New Revision: 1846988
URL: http://svn.apache.org/viewvc?rev=1846988=rev
Log:
OAK-7909: Backport and validate OAK-7867 to Oak 1.6
Merged r1845415
Modified:
jackrabbit/oak/branches/1.8/ (props changed)
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Propchange: jackrabbit/oak/branches/1.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Nov 20 10:43:49 2018
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911
,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1846057
+/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911
,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1845415,1846057
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java?rev=1846988=1846987=1846988=diff
==
---
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
(original)
+++
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Tue Nov 20 10:43:49 2018
@@ -332,7 +332,14 @@ public class FileStoreIT {
private int readOrEnd
svn commit: r1846986 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/ oak-segment
Author: mduerig Date: Tue Nov 20 10:41:16 2018 New Revision: 1846986 URL: http://svn.apache.org/viewvc?rev=1846986=rev Log: OAK-7909: Backport and validate OAK-7867 to Oak 1.6 Merged r1845405 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriterBuilder.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriteOperationHandler.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPoolTest.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Nov 20 10:41:16 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1846057 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1845405,1846057 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak
svn commit: r1846985 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig Date: Tue Nov 20 09:51:18 2018 New Revision: 1846985 URL: http://svn.apache.org/viewvc?rev=1846985=rev Log: OAK-7909: Backport and validate OAK-7867 to Oak 1.6 Merged r1845135 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Nov 20 09:51:18 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845336,1846057 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822642,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826833,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911 ,1830923,1831157-1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836167-1836168,1836170-1836187,1836189-1836196,1836206,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840462,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844625,1844627,1844642,1844728,1844775,1844932,1845135,1845336,1846057 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java?rev=1846985=1846984=1846985=diff == --- jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java (original) +++ jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Tue Nov 20 09:51:18 2018 @@ -19,14 +19,19 @@ package org.apache.jackrabbit.oak.segment.file; import static com.google.common.collect.Maps.newLinkedHashMap
svn commit: r1845994 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Author: mduerig
Date: Wed Nov 7 09:15:42 2018
New Revision: 1845994
URL: http://svn.apache.org/viewvc?rev=1845994=rev
Log:
OAK-7885: Performance regression in FlatTreeUpdateTest
Break out early when purging the tracked commits.
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java?rev=1845994=1845993=1845994=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Wed Nov 7 09:15:42 2018
@@ -23,6 +23,7 @@ import static com.google.common.collect.
import static com.google.common.collect.Queues.newConcurrentLinkedQueue;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentHashMap;
@@ -83,7 +84,17 @@ class CommitsTracker {
public void trackExecutedCommitOf(Thread thread) {
long t = System.currentTimeMillis();
-commits.removeIf(c -> c.timeStamp < t - 6);
+final Iterator it = commits.iterator();
+
+// Purge the queue
+// Avoiding removeIf allows us to bail out early. See OAK-7885
+while (it.hasNext()) {
+if (it.next().timeStamp < t - 6) {
+it.remove();
+} else {
+break;
+}
+}
commits.offer(new Commit(t, thread.getName()));
}
svn commit: r1845902 - in /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment: SegmentCache.java file/FileStoreUtil.java file/GCJournal.java file/Reclaimers.java file
Author: mduerig
Date: Tue Nov 6 11:54:10 2018
New Revision: 1845902
URL: http://svn.apache.org/viewvc?rev=1845902=rev
Log:
@trivial: broken Javadoc links
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreUtil.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GCJournal.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Reclaimers.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/TarRevisions.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java?rev=1845902=1845901=1845902=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
Tue Nov 6 11:54:10 2018
@@ -42,9 +42,9 @@ import org.jetbrains.annotations.NotNull
* SegmentId#isBulkSegmentId() bulk} segments.
*
* Conceptually this cache serves as a 2nd level cache for segments. The 1st
- * level cache is implemented by memoising the segment in its id (see {@link
+ * level cache is implemented by memoising the segment in its id (see {@code
* SegmentId#segment}. Every time an segment is evicted from this cache the
- * memoised segment is discarded (see {@link SegmentId#onAccess}.
+ * memoised segment is discarded (see {@code SegmentId#onAccess}.
*/
public abstract class SegmentCache {
@@ -105,7 +105,7 @@ public abstract class SegmentCache {
/**
* Record a hit in this cache's underlying statistics.
*
- * @see SegmentId#onAccess
+ * See {@code SegmentId#onAccess}
*/
public abstract void recordHit();
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreUtil.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreUtil.java?rev=1845902=1845901=1845902=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreUtil.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreUtil.java
Tue Nov 6 11:54:10 2018
@@ -40,7 +40,7 @@ class FileStoreUtil {
*
* @param store An instance of {@link SegmentStore}.
* @param idProvider The {@code SegmentIdProvider} of the {@code store}
- * @param journal Path to the journal file.
+ * @param journalFile The journal of the {@code store}
* @return An instance of {@link RecordId}, or {@code null} if none could
be
* found.
* @throws IOException If an I/O error occurs.
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GCJournal.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GCJournal.java?rev=1845902=1845901=1845902=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GCJournal.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/GCJournal.java
Tue Nov 6 11:54:10 2018
@@ -29,9 +29,8 @@ import java.util.List;
import com.google.common.base.Joiner;
import org.apache.jackrabbit.oak.segment.RecordId;
-import org.apache.jackrabbit.oak.segment.spi.persistence.GCJournalFile;
import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
-import org.apache.jackrabbit.oak.segment.file.tar.TarPersistence;
+import org.apache.jackrabbit.oak.segment.spi.persistence.GCJournalFile;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
@@ -39,7 +38,7 @@ import org.slf4j.LoggerFactory;
/**
* Persists the repository size and the reclaimed size following a cleanup
- * operation in the {@link TarPersistence#GC_JOURNAL gc journal} file with the
format:
+ * operation in the {@code gc.log} file with the format:
* 'repoSize, reclaimedSize, timestamp, gc generation, gc full generation
(since Oak 1.8),
* number of nodes compacted, root id (since Oak 1.8)'.
*/
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Reclaimers.java
URL
svn commit: r1845901 - in /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment: spi/persistence/SegmentNodeStorePersistence.java split/SplitSegmentArchiveManager.java
Author: mduerig
Date: Tue Nov 6 11:54:03 2018
New Revision: 1845901
URL: http://svn.apache.org/viewvc?rev=1845901=rev
Log:
@trivial: typo
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentNodeStorePersistence.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/split/SplitSegmentArchiveManager.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentNodeStorePersistence.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentNodeStorePersistence.java?rev=1845901=1845900=1845901=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentNodeStorePersistence.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentNodeStorePersistence.java
Tue Nov 6 11:54:03 2018
@@ -18,11 +18,11 @@
*/
package org.apache.jackrabbit.oak.segment.spi.persistence;
+import java.io.IOException;
+
import org.apache.jackrabbit.oak.segment.spi.monitor.FileStoreMonitor;
import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
-import java.io.IOException;
-
/**
* This type is a main entry point for the segment node store persistence. It's
* used every time the access to the underlying storage (eg. tar files) is
required.
@@ -35,7 +35,7 @@ public interface SegmentNodeStorePersist
*
* @param memoryMapping whether the memory mapping should be used (if the
given
* persistence supports it)
- * @param offHeapAccess whether off heap access for segements should be
used
+ * @param offHeapAccess whether off heap access for segments should be used
* @param ioMonitor object used to monitor segment-related IO access. The
* implementation should call the appropriate methods when
* accessing segments.
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/split/SplitSegmentArchiveManager.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/split/SplitSegmentArchiveManager.java?rev=1845901=1845900=1845901=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/split/SplitSegmentArchiveManager.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/split/SplitSegmentArchiveManager.java
Tue Nov 6 11:54:03 2018
@@ -16,12 +16,6 @@
*/
package org.apache.jackrabbit.oak.segment.split;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveManager;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveReader;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveWriter;
-import org.jetbrains.annotations.NotNull;
-import org.jetbrains.annotations.Nullable;
-
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
@@ -29,6 +23,12 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.UUID;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveManager;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveReader;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveWriter;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
public class SplitSegmentArchiveManager implements SegmentArchiveManager {
private final SegmentArchiveManager roArchiveManager;
@@ -116,7 +116,7 @@ public class SplitSegmentArchiveManager
if (roArchiveList.contains(to)) {
throw new IOException("Can't overwrite the read-only " + to);
} else if (roArchiveList.contains(from)) {
-throw new IOException("Can't copy the archive between persistances
" + from + " -> " + to);
+throw new IOException("Can't copy the archive between persistence
" + from + " -> " + to);
} else {
rwArchiveManager.copyFile(from, to);
}
svn commit: r1845415 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig
Date: Thu Nov 1 08:53:05 2018
New Revision: 1845415
URL: http://svn.apache.org/viewvc?rev=1845415=rev
Log:
OAK-7867: Flush thread gets stuck when input stream of binaries block
Improve synchronisation of blocking blob in test
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java?rev=1845415=1845414=1845415=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Thu Nov 1 08:53:05 2018
@@ -295,7 +295,14 @@ public class FileStoreIT {
private int readOrEnd() {
if (blocking.get()) {
-reading = true;
+if (!reading) {
+readMonitor.enter();
+try {
+reading = true;
+} finally {
+readMonitor.leave();
+}
+}
return 0;
} else {
return -1;
svn commit: r1845405 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ test/java/org/apache/jackrabbit/oak/segment/ test/java/org/apache/jackrabbit/oak/segme
Author: mduerig
Date: Thu Nov 1 08:26:51 2018
New Revision: 1845405
URL: http://svn.apache.org/viewvc?rev=1845405=rev
Log:
OAK-7867: Flush thread gets stuck when input stream of binaries block
Push write operations into the segment writer
Generalise handling of GCGeneration
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriterBuilder.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriteOperationHandler.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPoolTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java?rev=1845405=1845404=1845405=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Thu Nov 1 08:26:51 2018
@@ -65,6 +65,7 @@ import org.apache.jackrabbit.oak.api.Blo
import org.apache.jackrabbit.oak.api.PropertyState;
import org.apache.jackrabbit.oak.api.Type;
import org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState;
+import org.apache.jackrabbit.oak.segment.RecordWriters.RecordWriter;
import org.apache.jackrabbit.oak.segment.WriteOperationHandler.WriteOperation;
import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
@@ -140,107 +141,53 @@ public class DefaultSegmentWriter implem
@NotNull
RecordId writeMap(@Nullable final MapRecord base, @NotNull final
Map changes) throws IOException {
-return writeOperationHandler.execute(new SegmentWriteOperation() {
-
-@NotNull
-@Override
-public RecordId execute(@NotNull SegmentBufferWriter writer)
throws IOException {
-return with(writer).writeMap(base, changes);
-}
-
-});
+return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
+.writeMap(base, changes);
}
@NotNull
RecordId writeList(@NotNull final List list) throws IOException {
-return writeOperationHandler.execute(new SegmentWriteOperation() {
-
-@NotNull
-@Override
-public RecordId execute(@NotNull SegmentBufferWriter writer)
throws IOException {
-return with(writer).writeList(list);
-}
-
-});
+return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
+.writeList(list);
}
@NotNull
RecordId writeString(@NotNull final String string) throws IOException {
-return writeOperationHandler.execute(new SegmentWriteOperation() {
-
-@NotNull
-@Override
-public RecordId execute(@NotNull SegmentBufferWriter writer)
throws IOException {
-return with(writer).writeString(string);
-}
-
-});
+return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
+.writeString(string);
}
@Override
@NotNull
public RecordId writeBlob(@NotNull final Blob blob) throws IOException {
-return writeOperationHandler.execute(new SegmentWriteOperation() {
-
-@NotNull
-@Override
-public RecordId execute(@NotNull SegmentBufferWriter writer)
throws IOException {
-return with(writer).writeBlob(blob);
-}
-
-});
+return new
SegmentWriteOperation(writeOperationHandler.getGCGeneration())
+.writeBlob(blob);
}
@NotNull
RecordId writeBlock(@NotNull final byte[] bytes, final int offset, final
int length) throws IOException {
-return writeOperationHandler.execute(new SegmentWriteOperation() {
-
-@NotNull
-@Override
-public RecordId execute(@NotNull SegmentBufferWriter writer)
throws IOException {
-return with(writer).writeBlock(bytes, offset, length);
-}
-
-});
+return new
SegmentWriteOperation
svn commit: r1845242 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java oak-segment-tar/src/test/java/org/apache/jackrab
Author: mduerig Date: Tue Oct 30 14:05:17 2018 New Revision: 1845242 URL: http://svn.apache.org/viewvc?rev=1845242=rev Log: OAK-7853: SegmentBufferWriter not flushed after OnRC merged r1844549 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.6/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 14:05:17 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840226,1842677,1843 222,1843231,1844642,1844728 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840226,1842677,1843 222,1843231,1844549,1844642,1844728 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java URL: http
svn commit: r1845241 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java oak-segment-tar/src/test/java/org/apache/jackrab
Author: mduerig Date: Tue Oct 30 13:59:46 2018 New Revision: 1845241 URL: http://svn.apache.org/viewvc?rev=1845241=rev Log: OAK-7853: SegmentBufferWriter not flushed after OnRC merged r1844549 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 13:59:46 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844642,1844728,1844932 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844549,1844642,1844728,1844932 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1845241=1845240=1845241=diff == --- jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java (original) +++ jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java Tue Oct 30 13:59:46 2018 @@ -68,6 +68,11 @@ public class SegmentBufferWriterPool imp */ private final Set disposed = newHashSet(); +/** + * Retired writers that have not yet been flushed from a previous GC generation + */ +private
svn commit: r1845234 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ test/java/org/apache/jackrabbit/oak/segment/ test/java/org/apache/jackrabbit/oak/segme
Author: mduerig
Date: Tue Oct 30 12:58:57 2018
New Revision: 1845234
URL: http://svn.apache.org/viewvc?rev=1845234=rev
Log:
OAK-7838: oak-run check crashes JVM
Do away with the background thread and piggy back purge operations instead
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreStats.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CommitsTrackerTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java?rev=1845234=1845233=1845234=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/CommitsTracker.java
Tue Oct 30 12:58:57 2018
@@ -19,17 +19,17 @@
package org.apache.jackrabbit.oak.segment;
-import static java.util.concurrent.TimeUnit.MINUTES;
+import static com.google.common.collect.Maps.newHashMap;
+import static com.google.common.collect.Queues.newConcurrentLinkedQueue;
-import java.io.Closeable;
import java.util.HashMap;
import java.util.Map;
+import java.util.Queue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.stream.Stream;
import com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
-import org.apache.jackrabbit.oak.segment.file.Scheduler;
/**
* A simple tracker for the source of commits (writes) in
@@ -42,26 +42,28 @@ import org.apache.jackrabbit.oak.segment
*
* This class delegates thread-safety to its underlying state variables.
*/
-class CommitsTracker implements Closeable {
-private final boolean collectStackTraces;
+class CommitsTracker {
private final String[] threadGroups;
+private final int otherWritersLimit;
+private final boolean collectStackTraces;
private final ConcurrentMap queuedWritersMap;
-private final ConcurrentMap commitsCountPerThreadGroup;
-private final ConcurrentMap commitsCountOtherThreads;
-private final ConcurrentMap
commitsCountPerThreadGroupLastMinute;
-private final Scheduler commitsTrackerScheduler = new
Scheduler("CommitsTracker background tasks");
+private final Queue commits = newConcurrentLinkedQueue();
+
+private static final class Commit {
+final long timeStamp;
+final String thread;
+
+Commit(long timeStamp, String thread) {
+this.timeStamp = timeStamp;
+this.thread = thread;
+}
+}
CommitsTracker(String[] threadGroups, int otherWritersLimit, boolean
collectStackTraces) {
this.threadGroups = threadGroups;
+this.otherWritersLimit = otherWritersLimit;
this.collectStackTraces = collectStackTraces;
-this.commitsCountPerThreadGroup = new ConcurrentHashMap<>();
-this.commitsCountPerThreadGroupLastMinute = new ConcurrentHashMap<>();
-this.commitsCountOtherThreads = new
ConcurrentLinkedHashMap.Builder()
-.maximumWeightedCapacity(otherWritersLimit).build();
this.queuedWritersMap = new ConcurrentHashMap<>();
-
-commitsTrackerScheduler.scheduleWithFixedDelay("TarMK commits tracker
stats resetter", 1, MINUTES,
-this::resetStatistics);
}
public void trackQueuedCommitOf(Thread t) {
@@ -79,23 +81,23 @@ class CommitsTracker implements Closeabl
queuedWritersMap.remove(t.getName());
}
-public void trackExecutedCommitOf(Thread t) {
-String group = findGroupFor(t);
-
-if (group.equals("other")) {
-commitsCountOtherThreads.compute(t.getName(), (w, v) -> v == null
? 1 : v + 1);
-}
+public void trackExecutedCommitOf(Thread thread) {
+long t = System.currentTimeMillis();
+commits.removeIf(c -> c.timeStamp < t - 6);
+commits.offer(new Commit(t, thread.getName()));
+}
-commitsCountPerThreadGroup.compute(group, (w, v) -> v == null ? 1 : v
+ 1);
+public Map getQueuedWritersMap() {
+return new HashMap<>(queuedWritersMap);
}
-private String findGroupFor(Thread t) {
+private String findGroupFor(String thread) {
if (threadGroups == null) {
return "other";
}
-
+
for (String group : threadGroups) {
-
svn commit: r1845226 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Author: mduerig Date: Tue Oct 30 11:34:37 2018 New Revision: 1845226 URL: http://svn.apache.org/viewvc?rev=1845226=rev Log: OAK-7854: Add liveliness monitoring for FileStore background operations merged r1844642 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 11:34:37 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844728,1844932 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844642,1844728,1844932 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1845226=1845225=1845226=diff == --- jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java (original) +++ jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java Tue Oct 30 11:34:37 2018 @@ -42,6 +42,7 @@ import static org.apache.jackrabbit.oak. import static org.apache.jackrabbit.oak.segment.file.Reclaimers.newOldReclaimer; import static org.apache.jackrabbit.oak.segment.file.TarRevisions.EXPEDITE_OPTION; import static org.apache.jackrabbit.oak.segment.file.TarRevisions.timeout; +import static org.apache.jackrabbit.oak.stats.StatsOptions.METRICS_ONLY; import java.io.File
svn commit: r1845225 - in /jackrabbit/oak/branches/1.4: ./ oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java
Author: mduerig Date: Tue Oct 30 11:29:13 2018 New Revision: 1845225 URL: http://svn.apache.org/viewvc?rev=1845225=rev Log: OAK-7854: Add liveliness monitoring for FileStore background operations merged r1844642 Modified: jackrabbit/oak/branches/1.4/ (props changed) jackrabbit/oak/branches/1.4/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java Propchange: jackrabbit/oak/branches/1.4/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 11:29:13 2018 @@ -1,4 +1,4 @@ /jackrabbit/oak/branches/1.0:1665962 /jackrabbit/oak/branches/1.6:1802566,1819951,1819977,1830228,1835110 -/jackrabbit/oak/trunk:1733615,1733875,1733913,1733929,1734230,1734254,1734279,1734941,1735052,1735081,1735109,1735141,1735267,1735405,1735484,1735549,1735564,1735588,1735622,1735638,1735919,1735983,1736176,1737309-1737310,1737334,1737349,1737998,1738004,1738136,1738138,1738207,1738234,1738252,1738775,1738795,1738833,1738950,1738957,1738963,1739712,1739760,1739867,1739894,1739959-1739960,1740114,1740116,1740250,1740333,1740349,1740360,1740625-1740626,1740774,1740837,1740879,1740971,1741016,1741032,1741339,1741343,1742077,1742117,1742125,1742363,1742520,1742888,1742916,1743097,1743172,1743343,1743674,1744265,1744292,1744589,1744670,1744672,1744959,1745038,1745127,1745197,1745336,1745368,1746086,1746117,1746342,1746345,1746408,1746634,1746696,1746981,1747198,1747200,1747341-1747342,1747380,1747387,1747406,1747492,1747512,1747654,1748505,1748553,1748722,1748870,1749275,1749350,1749424,1749443,1749464,1749475,1749645,1749662,1749815,1749872,1749875,1749899,1750052,1750076-1750077,1750287 ,1750457,1750462,1750465,1750495,1750626,1750656,1750809,1750886-1750887,1751396,1751410,1751419,1751445-1751446,1751478,1751748,1751753,1751755,1751871,1752198,1752202,1752259,1752273-1752274,1752283,1752292,1752438,1752447-1752448,1752508,1752596,1752616,1752659,1752672,1753262,1753331-1753332,1753335-1753336,1753355,1753444,1753481,1754117,1754239,1755157,1755191,1756505-1756506,1756520,1756580,1757119,1757166,1758213,1758713,1759433,1759754,1759795,1759826,1760326,1760340,1760373,1760387,1760486,1760492,1760494,1760661-1760662,1760677,1760701,1760709,1760946,1761412,1761444,1761571,1761762,1761787,1761866,1761876,1762453,1762463,1762612,1762632,1762635,1762825,1763347,1763355-1763356,1763378,1763465,1763735,1764475,1764678,1764705,1764814,1764898,1765817,1765983,1766071,1766390,1766423,1766496,1766519,1766554,1766644,1767025,1767265,1767502,1767704,1768446,1768637,1769078,1769939-1769940,1770694,1770982,1771022,1771093,1771098,1771739,1771852,1771870,1771902,1772155,1772162,1772 228,1772593,1772768,1772906,1773190,1774141,1774256,1774445,1774497,1774519,1774787,1775474,1775622,1775628,1775757,1778112,1778423,1778968,1779137,1779478,1780388,1780424,1780538,1780543,1781068,1781075,1781386,1781846,1781907,1782476,1782966,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783738,1783773,1783855,1783891,1784023,1784034,1784130,1784251,1784551,1784574,1784689,1785161,1785172,1785283,1785838,1785946,1787074,1787217,1787425,1789056,1792463,1792742,1793013,1793088,1793644,1795314,1795330,1795475,1795488,1795491,1795613,1795618,1796144,1798035,1798832,1798834,1799219,1799389,1799924,1800974,1801011,1801013,1802548,1802973,1803026,1804437,1807308,1808125,1808128,1808142,1808240,1808246,1809024,1809026,1809163,1809745,1811380,1811655,1811952,1811963,1811986,1813538,1814189,1814332,1814397,1815438,1817326,1818645,1819048,1819050,1821325,1821516,1823172,1823655,1826237,1826640,1826932,1826957,1828502,1829527,1829587,1829665,1829987,1830019,1830160,1830239,1831190,1 831374,1833308,1834648-1834649,1834681,1835060,1837475,1837998,1838637 +/jackrabbit/oak/trunk:1733615,1733875,1733913,1733929,1734230,1734254,1734279,1734941,1735052,1735081,1735109,1735141,1735267,1735405,1735484,1735549,1735564,1735588,1735622,1735638,1735919,1735983,1736176,1737309-1737310,1737334,1737349,1737998,1738004,1738136,1738138,1738207,1738234,1738252,1738775,1738795,1738833,1738950,1738957,1738963,1739712,1739760,1739867,1739894,1739959-1739960,1740114,1740116,1740250,1740333,1740349,1740360,1740625-1740626,1740774,1740837,1740879,1740971,1741016,1741032,1741339,1741343,1742077,1742117,1742125,1742363,1742520,1742888,1742916,1743097,1743172,1743343,1743674,1744265,1744292,1744589,1744670,1744672,1744959,1745038,1745127,1745197,1745336,1745368,1746086,1746117,1746342,1746345,1746408,1746634,1746696,1746981,1747198,1747200,1747341-1747342,1747380,1747387,1747406,1747492,1747512,1747654,1748505,1748553,1748722,1748870,1749275,1749350,1749424,1749443,1749464,1749475,1749645,1749662,1749815,1749872,1749875,1749899,1750052,1750076-1750077,1750287 ,1750457,1750462,1750465,1750495,1750626,1750656,1750809,1750886-1750887,1751396,1751410,1751419,1751445-1751446,1751478,1751748,1751753,1751755,1751871,1752198,1752202,1752259,1752273
svn commit: r1845224 - in /jackrabbit/oak/branches/1.6: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Author: mduerig Date: Tue Oct 30 11:23:11 2018 New Revision: 1845224 URL: http://svn.apache.org/viewvc?rev=1845224=rev Log: OAK-7854: Add liveliness monitoring for FileStore background operations merged r1844642 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 11:23:11 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840226,1842677,1843 222,1843231,1844728 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1836082,1837475,1837998,1838637,1839746,1840024,1840226,1842677,1843 222,1843231,1844642,1844728 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1845224
svn commit: r1845216 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
Author: mduerig Date: Tue Oct 30 10:25:42 2018 New Revision: 1845216 URL: http://svn.apache.org/viewvc?rev=1845216=rev Log: OAK-7837: oak-run check crashes with SNFE merged r1844325 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 30 10:25:42 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844728,1844932 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821027,1821130,1821140-1821141,1821178,1821237,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821617,1821663,1821665,1821668,1821681,1821847,1821975-1821983,1822121,1822201,1822207,1822527,1822723,1822808,1822850,1822934,1823135,1823163,1823169,1823172,1823655,1823669,1824196,1824198,1824253,1824255,1824896,1824962,1825065,1825362,1825381,1825442,1825448,1825466,1825470-1825471,1825475,1825523,1825525,1825561,1825619-1825621,1825651,1825654,1825992,1826079,1826090,1826096,1826216,1826237,1826338,1826516,1826532,1826551,1826560,1826638,1826640,1826730,1826932,1826957,1827423,1827472,1827486,1827816,1827977,1828349,1828439,1828502,1828529,1828948,1829527,1829534,1829546,1829569,1829587,1829665,1829854,1829864,1829978,1829985,1829987,1829998,1830019,1830048,1830160,1830171,1830197,1830209,1830239,1830347,1830748,1830911,1830923,1831157 -1831158,1831163,1831190,1831374,1831560,1831689,1832258,1832376,1832379,1832535,1833308,1833347,1833833,1834112,1834117,1834287,1834291,1834302,1834326,1834328,1834336,1834428,1834468,1834483,1834610,1834648-1834649,1834681,1834823,1834857-1834858,1835060,1835518,1835521,1835635,1835642,1835780,1835819,1836082,1836121,1836487,1836493,1837057,1837274,1837296,1837326,1837475,1837503,1837547,1837569,1837600,1837657,1837718,1837998,1838076,1838637,1839549,1839570,1839637,1839746,1840019,1840024,1840031,1840226,1840455,1840574,1841314,1841352,1842089,1842677,1843222,1843231,1843398,1843618,1843652,1843911,1844325,1844728,1844932 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java?rev=1845216=1845215=1845216=diff == --- jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java (original) +++ jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java Tue Oct 30 10:25:42 2018 @@ -53,6 +53,7 @@ import org.apache.jackrabbit.oak.api.Typ import org.apache.jackrabbit.oak.segment.SegmentBlob; import org.apache.jackrabbit.oak.segment.SegmentNodeStore; import org.apache.jackrabbit.oak.segment.SegmentNodeStoreBuilders; +import org.apache.jackrabbit.oak.segment.SegmentNotFoundException; import
svn commit: r1845135 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Author: mduerig
Date: Mon Oct 29 14:53:43 2018
New Revision: 1845135
URL: http://svn.apache.org/viewvc?rev=1845135=rev
Log:
OAK-7867: Flush thread gets stuck when input stream of binaries block
@Ignored test case
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java?rev=1845135=1845134=1845135=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Mon Oct 29 14:53:43 2018
@@ -19,21 +19,34 @@
package org.apache.jackrabbit.oak.segment.file;
import static com.google.common.collect.Maps.newLinkedHashMap;
+import static java.util.concurrent.Executors.newSingleThreadExecutor;
+import static java.util.concurrent.TimeUnit.SECONDS;
import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.fileStoreBuilder;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
+import java.io.InputStream;
import java.io.RandomAccessFile;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Random;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import com.google.common.util.concurrent.Monitor;
+import com.google.common.util.concurrent.Monitor.Guard;
import org.apache.jackrabbit.oak.api.Blob;
+import org.apache.jackrabbit.oak.plugins.memory.AbstractBlob;
import org.apache.jackrabbit.oak.plugins.memory.ArrayBasedBlob;
import org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState;
import org.apache.jackrabbit.oak.segment.DefaultSegmentWriter;
@@ -44,6 +57,8 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
import org.apache.jackrabbit.oak.spi.state.ChildNodeEntry;
import org.apache.jackrabbit.oak.spi.state.NodeState;
+import org.jetbrains.annotations.NotNull;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -237,4 +252,102 @@ public class FileStoreIT {
}
}
+@Ignore("OAK-7867")
+@Test
+public void blockingBlob() throws Exception {
+
+/* A blob that blocks on read until unblocked */
+class BlockingBlob extends AbstractBlob {
+private final AtomicBoolean blocking = new AtomicBoolean(true);
+private final Monitor readMonitor = new Monitor();
+private boolean reading = false;
+
+public boolean waitForRead(int time, TimeUnit unit) throws
InterruptedException {
+readMonitor.enter();
+try {
+return readMonitor.waitFor(new Guard(readMonitor) {
+@Override
+public boolean isSatisfied() {
+return reading;
+}
+}, time, unit);
+} finally {
+readMonitor.leave();
+}
+}
+
+public void unblock() {
+blocking.set(false);
+}
+
+@NotNull
+@Override
+public InputStream getNewStream() {
+return new InputStream() {
+
+@Override
+public int read() throws IOException {
+return readOrEnd();
+}
+
+@Override
+public int read(@NotNull byte[] b, int off, int len)
throws IOException {
+return readOrEnd();
+}
+
+private int readOrEnd() {
+if (blocking.get()) {
+reading = true;
+return 0;
+} else {
+return -1;
+}
+}
+};
+}
+
+@Override
+public long length() {
+return 1;
+}
+}
+
+ExecutorService upda
svn commit: r1844642 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Author: mduerig
Date: Tue Oct 23 12:47:38 2018
New Revision: 1844642
URL: http://svn.apache.org/viewvc?rev=1844642=rev
Log:
OAK-7854: Add liveliness monitoring for FileStore background operations
Expose a timer for monitoring the flush rate
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1844642=1844641=1844642=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Tue Oct 23 12:47:38 2018
@@ -25,6 +25,7 @@ import static org.apache.jackrabbit.oak.
import static
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.EMPTY_NODE;
import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static
org.apache.jackrabbit.oak.segment.file.PrintableBytes.newPrintableBytes;
+import static org.apache.jackrabbit.oak.stats.StatsOptions.METRICS_ONLY;
import java.io.IOException;
import java.nio.ByteBuffer;
@@ -52,6 +53,8 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.spi.persistence.RepositoryLock;
import
org.apache.jackrabbit.oak.segment.spi.persistence.SegmentNodeStorePersistence;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
+import org.apache.jackrabbit.oak.stats.TimerStats;
+import org.apache.jackrabbit.oak.stats.TimerStats.Context;
import org.jetbrains.annotations.NotNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -178,8 +181,15 @@ public class FileStore extends AbstractF
this.snfeListener = builder.getSnfeListener();
-fileStoreScheduler.scheduleWithFixedDelay(format("TarMK flush [%s]",
directory), 5, SECONDS,
- this::tryFlush);
+TimerStats flushTimer =
builder.getStatsProvider().getTimer("oak.segment.flush", METRICS_ONLY);
+fileStoreScheduler.scheduleWithFixedDelay(format("TarMK flush [%s]",
directory), 5, SECONDS, () -> {
+Context timer = flushTimer.time();
+try {
+tryFlush();
+} finally {
+timer.stop();
+}
+});
fileStoreScheduler.scheduleWithFixedDelay(format("TarMK filer reaper
[%s]", directory), 5, SECONDS,
fileReaper::reap);
svn commit: r1844549 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java test/java/org/apache/jackrabbit/oak/segment/file/FileStore
Author: mduerig
Date: Mon Oct 22 12:26:14 2018
New Revision: 1844549
URL: http://svn.apache.org/viewvc?rev=1844549=rev
Log:
OAK-7853: SegmentBufferWriter not flushed after OnRC
Test case and added a fix that flushes SBW instances disposed after OnRC
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java?rev=1844549=1844548=1844549=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java
Mon Oct 22 12:26:14 2018
@@ -67,6 +67,11 @@ public class SegmentBufferWriterPool imp
*/
private final Set disposed = newHashSet();
+/**
+ * Retired writers that have not yet been flushed from a previous GC
generation
+ */
+private final Set disposedOldGen = newHashSet();
+
@NotNull
private final SegmentIdProvider idProvider;
@@ -115,6 +120,11 @@ public class SegmentBufferWriterPool imp
toFlush.addAll(writers.values());
writers.clear();
+// Collect all writers from old GC generations that
+// have been disposed
+toFlush.addAll(disposedOldGen);
+disposedOldGen.clear();
+
// Collect all borrowed writers, which we need to wait for.
// Clear the list so they will get disposed once returned.
toReturn.addAll(borrowed);
@@ -191,7 +201,7 @@ public class SegmentBufferWriterPool imp
gcGeneration.get()
);
} else if (!writer.getGCGeneration().equals(gcGeneration.get())) {
-disposed.add(writer);
+disposedOldGen.add(writer);
writer = new SegmentBufferWriter(
idProvider,
reader,
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java?rev=1844549=1844548=1844549=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreIT.java
Mon Oct 22 12:26:14 2018
@@ -18,21 +18,32 @@
*/
package org.apache.jackrabbit.oak.segment.file;
+import static com.google.common.collect.Maps.newLinkedHashMap;
+import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.fileStoreBuilder;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import java.io.ByteArrayInputStream;
import java.io.File;
+import java.io.IOException;
import java.io.RandomAccessFile;
+import java.util.Map;
+import java.util.Map.Entry;
import java.util.Random;
+import java.util.concurrent.CountDownLatch;
import org.apache.jackrabbit.oak.api.Blob;
import org.apache.jackrabbit.oak.plugins.memory.ArrayBasedBlob;
+import org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState;
+import org.apache.jackrabbit.oak.segment.DefaultSegmentWriter;
import org.apache.jackrabbit.oak.segment.RecordId;
import org.apache.jackrabbit.oak.segment.SegmentNodeBuilder;
import org.apache.jackrabbit.oak.segment.SegmentNodeState;
import org.apache.jackrabbit.oak.segment.SegmentTestConstants;
+import org.apache.jackrabbit.oak.segment.file.tar.GCGeneration;
+import org.apache.jackrabbit.oak.spi.state.ChildNodeEntry;
+import org.apache.jackrabbit.oak.spi.state.NodeState;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -173,4 +184,57 @@ public class FileStoreIT {
}
}
+@Test
+public void snfeAfterOnRC()
+throws IOException, InvalidFileStoreVersionException, InterruptedException
{
+Map roots = newLinkedHashMap();
+try (FileStore rwStore =
fileStoreBuilder(getFileStoreFolder()).build()) {
+
+// Block scheduled journal updates
+CountDownLatch blockJournalUpdates = new CountDownLatch(1);
+
+// Ensure we have a non empty journal
svn commit: r1844342 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
Author: mduerig
Date: Fri Oct 19 13:15:00 2018
New Revision: 1844342
URL: http://svn.apache.org/viewvc?rev=1844342=rev
Log:
OAK-7838: oak-run check crashes JVM
@Ignored test case
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java?rev=1844342=1844341=1844342=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tooling/CheckInvalidRepositoryTest.java
Fri Oct 19 13:15:00 2018
@@ -19,15 +19,22 @@
package org.apache.jackrabbit.oak.segment.file.tooling;
import java.io.File;
+import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.HashSet;
import java.util.LinkedHashSet;
import java.util.Set;
+import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
+import org.apache.commons.io.FileUtils;
+import org.apache.jackrabbit.oak.segment.file.JournalEntry;
+import org.apache.jackrabbit.oak.segment.file.JournalReader;
+import org.apache.jackrabbit.oak.segment.file.tar.LocalJournalFile;
import org.apache.jackrabbit.oak.segment.tool.Check;
import org.junit.Before;
+import org.junit.Ignore;
import org.junit.Test;
/**
@@ -210,4 +217,46 @@ public class CheckInvalidRepositoryTest
assertExpectedOutput(strErr.toString(), Lists.newArrayList(
"Error while traversing /b:
java.lang.IllegalArgumentException: Segment reference out of bounds"));
}
+
+@Ignore("OAK-7838")
+@Test
+public void testLargeJournal() throws IOException {
+StringWriter strOut = new StringWriter();
+StringWriter strErr = new StringWriter();
+
+PrintWriter outWriter = new PrintWriter(strOut, true);
+PrintWriter errWriter = new PrintWriter(strErr, true);
+
+File segmentStoreFolder = new
File(temporaryFolder.getRoot().getAbsolutePath());
+File journalFile = new File(segmentStoreFolder, "journal.log");
+File largeJournalFile = temporaryFolder.newFile("journal.log.large");
+
+JournalReader journalReader = new JournalReader(new
LocalJournalFile(journalFile));
+JournalEntry journalEntry = journalReader.next();
+
+String journalLine = journalEntry.getRevision() + " root " +
journalEntry.getTimestamp() + "\n";
+
+for (int k = 0; k < 1; k++) {
+FileUtils.writeStringToFile(largeJournalFile, journalLine, true);
+}
+
+Check.builder()
+.withPath(segmentStoreFolder)
+.withJournal("journal.log.large")
+.withDebugInterval(Long.MAX_VALUE)
+.withCheckBinaries(true)
+.withCheckHead(true)
+.withFilterPaths(ImmutableSet.of("/"))
+.withCheckpoints(checkpoints)
+.withOutWriter(outWriter)
+.withErrWriter(errWriter)
+.build()
+.run();
+
+outWriter.close();
+errWriter.close();
+
+assertExpectedOutput(strOut.toString(), Lists.newArrayList("No good
revision found"));
+}
+
}
svn commit: r1844325 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
Author: mduerig
Date: Fri Oct 19 10:35:50 2018
New Revision: 1844325
URL: http://svn.apache.org/viewvc?rev=1844325=rev
Log:
OAK-7837: oak-run check crashes with SNFE
Catch SNFE to mark revision as invalid
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java?rev=1844325=1844324=1844325=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tooling/ConsistencyChecker.java
Fri Oct 19 10:35:50 2018
@@ -53,6 +53,7 @@ import org.apache.jackrabbit.oak.api.Typ
import org.apache.jackrabbit.oak.segment.SegmentBlob;
import org.apache.jackrabbit.oak.segment.SegmentNodeStore;
import org.apache.jackrabbit.oak.segment.SegmentNodeStoreBuilders;
+import org.apache.jackrabbit.oak.segment.SegmentNotFoundException;
import org.apache.jackrabbit.oak.segment.file.FileStore;
import org.apache.jackrabbit.oak.segment.file.FileStoreBuilder;
import org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException;
@@ -231,8 +232,8 @@ public class ConsistencyChecker implemen
if (overallValid) {
lastValidJournalEntry = journalEntry;
}
-} catch (IllegalArgumentException e) {
-checker.printError("Skipping invalid record id {0}",
revision);
+} catch (IllegalArgumentException | SegmentNotFoundException
e) {
+checker.printError("Skipping invalid record id {0}: {1}",
revision, e);
}
}
svn commit: r1842661 - in /jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure: AzureSegmentArchiveReader.java AzureSegmentArchiveWriter.java AzureUtilities.ja
Author: mduerig
Date: Tue Oct 2 22:07:47 2018
New Revision: 1842661
URL: http://svn.apache.org/viewvc?rev=1842661=rev
Log:
OAK-7770: Azure Segment Store: Store segments off heap when memory mapping is
disabled
Modified:
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveReader.java
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
Modified:
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveReader.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveReader.java?rev=1842661=1842660=1842661=diff
==
---
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveReader.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveReader.java
Tue Oct 2 22:07:47 2018
@@ -16,14 +16,9 @@
*/
package org.apache.jackrabbit.oak.segment.azure;
-import com.google.common.base.Stopwatch;
-import com.microsoft.azure.storage.StorageException;
-import com.microsoft.azure.storage.blob.CloudBlob;
-import com.microsoft.azure.storage.blob.CloudBlobDirectory;
-import com.microsoft.azure.storage.blob.CloudBlockBlob;
-import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveEntry;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveReader;
+import static java.lang.Boolean.getBoolean;
+import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.getSegmentFileName;
+import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.readBufferFully;
import java.io.File;
import java.io.IOException;
@@ -37,10 +32,17 @@ import java.util.UUID;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
-import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.getSegmentFileName;
-import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.readBufferFully;
+import com.google.common.base.Stopwatch;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobDirectory;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveEntry;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveReader;
public class AzureSegmentArchiveReader implements SegmentArchiveReader {
+static final boolean OFF_HEAP = getBoolean("access.off.heap");
private final CloudBlobDirectory archiveDirectory;
@@ -73,7 +75,13 @@ public class AzureSegmentArchiveReader i
if (indexEntry == null) {
return null;
}
-ByteBuffer buffer = ByteBuffer.allocate(indexEntry.getLength());
+
+ByteBuffer buffer;
+if (OFF_HEAP) {
+buffer = ByteBuffer.allocateDirect(indexEntry.getLength());
+} else {
+buffer = ByteBuffer.allocate(indexEntry.getLength());
+}
ioMonitor.beforeSegmentRead(pathAsFile(), msb, lsb,
indexEntry.getLength());
Stopwatch stopwatch = Stopwatch.createStarted();
readBufferFully(getBlob(getSegmentFileName(indexEntry)), buffer);
Modified:
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java?rev=1842661=1842660=1842661=diff
==
---
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
Tue Oct 2 22:07:47 2018
@@ -16,6 +16,7 @@
*/
package org.apache.jackrabbit.oak.segment.azure;
+import static
org.apache.jackrabbit.oak.segment.azure.AzureSegmentArchiveReader.OFF_HEAP;
import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.getSegmentFileName;
import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.readBufferFully;
@@ -108,7 +109,13 @@ public class AzureSegmentArchiveWriter i
if (indexEntry == null) {
return null;
}
-
svn commit: r1841460 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ main/java/org/apache/jackrabbit/oak/segment/file/ test/java/org/apache/jackrabbit/oak/
Author: mduerig
Date: Thu Sep 20 11:59:03 2018
New Revision: 1841460
URL: http://svn.apache.org/viewvc?rev=1841460=rev
Log:
OAK-7773: Implement monitoring for allocated byte buffers
Initial implementation
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitor.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitorTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractFileStore.java
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitor.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitor.java?rev=1841460=auto
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitor.java
(added)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferMonitor.java
Thu Sep 20 11:59:03 2018
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.jackrabbit.oak.segment;
+
+import static com.google.common.collect.Sets.newConcurrentHashSet;
+import static org.apache.jackrabbit.oak.stats.StatsOptions.METRICS_ONLY;
+
+import java.lang.ref.ReferenceQueue;
+import java.lang.ref.WeakReference;
+import java.nio.ByteBuffer;
+import java.util.Set;
+
+import org.apache.jackrabbit.oak.stats.CounterStats;
+import org.apache.jackrabbit.oak.stats.StatisticsProvider;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * This class exposes {@link CounterStats} for allocations and de-allocations
+ * of {@link ByteBuffer} instances:
+ *
+ * {@link #DIRECT_BUFFER_COUNT}: number of allocated direct byte
+ * buffers.
+ * {@link #DIRECT_BUFFER_CAPACITY}: total capacity of the allocated
+ * direct byte buffers.
+ * {@link #HEAP_BUFFER_COUNT}: number of allocated heap byte
+ * buffers.
+ * {@link #HEAP_BUFFER_CAPACITY}: total capacity of the allocated
+ * heap byte buffers.
+ *
+ *
+ * Users of this class call {@link #trackAllocation(ByteBuffer)} to update
above statistics.
+ */
+public class SegmentBufferMonitor {
+
+/**
+ * Number of allocated direct byte buffers
+ */
+public static final String DIRECT_BUFFER_COUNT =
"oak.segment.direct-buffer-count";
+
+/**
+ * Total capacity of the allocated direct byte buffers.
+ */
+public static final String DIRECT_BUFFER_CAPACITY =
"oak.segment.direct-buffer-capacity";
+
+/**
+ * Number of allocated heap byte buffers
+ */
+public static final String HEAP_BUFFER_COUNT =
"oak.segment.heap-buffer-count";
+
+/**
+ * Total capacity of the allocated heap byte buffers.
+ */
+public static final String HEAP_BUFFER_CAPACITY =
"oak.segment.heap-buffer-capacity";
+
+@NotNull
+private final Set buffers = newConcurrentHashSet();
+
+@NotNull
+private final ReferenceQueue referenceQueue = new
ReferenceQueue<>();
+
+@NotNull
+private final CounterStats directBufferCount;
+
+@NotNull
+private final CounterStats directBufferCapacity;
+
+@NotNull
+private final CounterStats heapBufferCount;
+
+@NotNull
+private final CounterStats heapBufferCapacity;
+
+/**
+ * Create a new instance using the passed {@code statisticsProvider} to
expose
+ * buffer allocations.
+ * @param statisticsProvider
+ */
+public SegmentBufferMonitor(@NotNull StatisticsProvider
statisticsProvider) {
+directBufferCount =
statisticsProvider.getCounterStats(DIRECT_BUFFER_COUNT, METRICS_ONLY);
+directBufferCapacity =
statisticsProvider.getCounterStats(DIRECT_BUFFER_CAPACITY, METRICS_ONLY);
+heapBufferCount =
statisticsProvider.getCounterStats(HEAP_BUFFER_COUNT, METRICS_ONLY);
+heapBufferCapacity =
statisticsProvider.getCounterStats(HEAP_BUFFER_CAPACITY,
svn commit: r1841443 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
Author: mduerig
Date: Thu Sep 20 08:50:04 2018
New Revision: 1841443
URL: http://svn.apache.org/viewvc?rev=1841443=rev
Log:
OAK-7771: Make mmap and segment cache size configurable in SegmentCompactionIT
Use -Dmmap and -Dsegment-cache to set
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java?rev=1841443=1841442=1841443=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
Thu Sep 20 08:50:04 2018
@@ -27,13 +27,16 @@ import static com.google.common.util.con
import static com.google.common.util.concurrent.Futures.dereference;
import static
com.google.common.util.concurrent.Futures.immediateCancelledFuture;
import static
com.google.common.util.concurrent.MoreExecutors.listeningDecorator;
+import static java.lang.Boolean.parseBoolean;
import static java.lang.Integer.MAX_VALUE;
+import static java.lang.Integer.getInteger;
import static java.lang.String.valueOf;
import static java.lang.System.getProperty;
import static java.util.concurrent.TimeUnit.DAYS;
import static java.util.concurrent.TimeUnit.MINUTES;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.apache.commons.lang.RandomStringUtils.randomAlphabetic;
+import static
org.apache.jackrabbit.oak.segment.SegmentCache.DEFAULT_SEGMENT_CACHE_MB;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.FULL;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.TAIL;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.defaultGCOptions;
@@ -138,6 +141,10 @@ public class SegmentCompactionIT {
private static final boolean ENABLED =
SegmentCompactionIT.class.getSimpleName().equals(getProperty("test"));
+private static final boolean MMAP = parseBoolean(getProperty("mmap",
"true"));
+
+private static final int SEGMENT_CACHE_SIZE = getInteger("segment-cache",
DEFAULT_SEGMENT_CACHE_MB);
+
private static final Logger LOG =
LoggerFactory.getLogger(SegmentCompactionIT.class);
private final MBeanServer mBeanServer =
ManagementFactory.getPlatformMBeanServer();
@@ -162,8 +169,8 @@ public class SegmentCompactionIT {
private volatile ListenableFuture compactor =
immediateCancelledFuture();
private volatile ReadWriteLock compactionLock = null;
-private volatile int maxReaders =
Integer.getInteger("SegmentCompactionIT.maxReaders", 10);
-private volatile int maxWriters =
Integer.getInteger("SegmentCompactionIT.maxWriters", 10);
+private volatile int maxReaders =
getInteger("SegmentCompactionIT.maxReaders", 10);
+private volatile int maxWriters =
getInteger("SegmentCompactionIT.maxWriters", 10);
private volatile long maxStoreSize = 2000L;
private volatile int maxBlobSize = 100;
private volatile int maxStringSize = 100;
@@ -255,7 +262,8 @@ public class SegmentCompactionIT {
MetricStatisticsProvider statisticsProvider = new
MetricStatisticsProvider(mBeanServer, executor);
FileStoreBuilder builder = fileStoreBuilder(folder.getRoot());
fileStore = builder
-.withMemoryMapping(true)
+.withMemoryMapping(MMAP)
+.withSegmentCacheSize(SEGMENT_CACHE_SIZE)
.withGCMonitor(gcMonitor)
.withGCOptions(gcOptions)
.withIOMonitor(new MetricsIOMonitor(statisticsProvider))
svn commit: r1841210 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
Author: mduerig
Date: Tue Sep 18 09:33:25 2018
New Revision: 1841210
URL: http://svn.apache.org/viewvc?rev=1841210=rev
Log:
OAK-7761: SegmentTarWriter#readSegment does not check the return value of
FileChannel#read
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java?rev=1841210=1841209=1841210=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
Tue Sep 18 09:33:25 2018
@@ -20,8 +20,10 @@ package org.apache.jackrabbit.oak.segmen
import static com.google.common.base.Charsets.UTF_8;
import static com.google.common.base.Preconditions.checkState;
+import static org.apache.jackrabbit.oak.commons.IOUtils.readFully;
import static
org.apache.jackrabbit.oak.segment.file.tar.TarConstants.BLOCK_SIZE;
+import java.io.EOFException;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
@@ -138,7 +140,9 @@ public class SegmentTarWriter implements
}
checkState(channel != null); // implied by entry != null
ByteBuffer data = ByteBuffer.allocate(indexEntry.getLength());
-channel.read(data, indexEntry.getPosition());
+if (readFully(channel, indexEntry.getPosition(), data) <
indexEntry.getLength()) {
+throw new EOFException();
+}
data.rewind();
return data;
}
svn commit: r1841211 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
Author: mduerig
Date: Tue Sep 18 09:33:32 2018
New Revision: 1841211
URL: http://svn.apache.org/viewvc?rev=1841211=rev
Log:
OAK-7762: Store segments off heap when memory mapping is disabled
Use -Daccess.off.heap=true to enable
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java?rev=1841211=1841210=1841211=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
Tue Sep 18 09:33:32 2018
@@ -19,6 +19,7 @@
package org.apache.jackrabbit.oak.segment.file.tar;
import static com.google.common.base.Preconditions.checkState;
+import static java.lang.Boolean.getBoolean;
import static java.nio.channels.FileChannel.MapMode.READ_ONLY;
import static org.apache.jackrabbit.oak.commons.IOUtils.readFully;
@@ -35,6 +36,8 @@ import java.nio.channels.FileChannel;
*/
abstract class FileAccess {
+private static final boolean OFF_HEAP = getBoolean("access.off.heap");
+
abstract boolean isMemoryMapped();
abstract int length() throws IOException;
@@ -113,7 +116,12 @@ abstract class FileAccess {
@Override
public synchronized ByteBuffer read(int position, int length)
throws IOException {
-ByteBuffer entry = ByteBuffer.allocate(length);
+ByteBuffer entry;
+if (OFF_HEAP) {
+entry = ByteBuffer.allocateDirect(length);
+} else {
+entry = ByteBuffer.allocate(length);
+}
if (readFully(channel, position, entry) < length) {
throw new EOFException();
}
svn commit: r1841209 - in /jackrabbit/oak/trunk: oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/ oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/ oak-segment-tar/src/main/java
Author: mduerig
Date: Tue Sep 18 09:33:19 2018
New Revision: 1841209
URL: http://svn.apache.org/viewvc?rev=1841209=rev
Log:
OAK-7760: Use NIO in the implementation of FileAccess.Mapped#read
Modified:
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/IOUtils.java
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/package-info.java
jackrabbit/oak/trunk/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/IOUtilsTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/FileAccess.java
Modified:
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/IOUtils.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/IOUtils.java?rev=1841209=1841208=1841209=diff
==
---
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/IOUtils.java
(original)
+++
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/IOUtils.java
Tue Sep 18 09:33:19 2018
@@ -22,6 +22,8 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
import java.util.Locale;
/**
@@ -44,7 +46,7 @@ public final class IOUtils {
* @param buffer the output buffer
* @param offthe offset in the buffer
* @param maxthe number of bytes to read at most
- * @return the number of bytes read, 0 meaning EOF
+ * @return the number of bytes read, 0 meaning EOF or no space in buffer
* @throws java.io.IOException If an error occurs.
*/
public static int readFully(InputStream in, byte[] buffer, int off, int
max) throws IOException {
@@ -61,6 +63,31 @@ public final class IOUtils {
}
return result;
}
+
+/**
+ * Try to read the given number of bytes starting at the specified position
+ * into the buffer. This method reads until the maximum number of bytes
have
+ * been read or until the end of the channel.
+ *
+ * @param channel the input channel
+ * @param positionthe position to start reading from the channel
+ * @param buffer the output buffer
+ * @return the number of bytes read, 0 meaning EOF or no space in buffer
+ * @throws java.io.IOException If an error occurs.
+ */
+public static int readFully(FileChannel channel, int position, ByteBuffer
buffer)
+throws IOException {
+int result = 0;
+while (buffer.remaining() > 0) {
+int count = channel.read(buffer, position);
+if (count < 0) {
+break;
+}
+result += count;
+position += count;
+}
+return result;
+}
/**
* Skip a number of bytes in an input stream.
Modified:
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/package-info.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/package-info.java?rev=1841209=1841208=1841209=diff
==
---
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/package-info.java
(original)
+++
jackrabbit/oak/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/package-info.java
Tue Sep 18 09:33:19 2018
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-@Version("1.1.1")
+@Version("1.2.1")
package org.apache.jackrabbit.oak.commons;
import org.osgi.annotation.versioning.Version;
Modified:
jackrabbit/oak/trunk/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/IOUtilsTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/IOUtilsTest.java?rev=1841209=1841208=1841209=diff
==
---
jackrabbit/oak/trunk/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/IOUtilsTest.java
(original)
+++
jackrabbit/oak/trunk/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/IOUtilsTest.java
Tue Sep 18 09:33:19 2018
@@ -16,12 +16,17 @@
*/
package org.apache.jackrabbit.oak.commons;
+import static org.apache.commons.io.FileUtils.writeByteArrayToFile;
+
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.EOFException;
+import java.io.File;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
import
svn commit: r1840540 - /jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Author: mduerig
Date: Tue Sep 11 11:19:48 2018
New Revision: 1840540
URL: http://svn.apache.org/viewvc?rev=1840540=rev
Log:
OAK-7708: Backport OAK-6648 to Oak 1.6
Merged r1808731
Modified:
jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Modified:
jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1840540=1840539=1840540=diff
==
---
jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/branches/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Tue Sep 11 11:19:48 2018
@@ -493,8 +493,8 @@ public class FileStore extends AbstractF
closeAndLogOnFail(closer);
// Try removing pending files in case the scheduler didn't have a
chance to run yet
-fileReaper.reap();
System.gc(); // for any memory-mappings that are no longer used
+fileReaper.reap();
log.info("TarMK closed: {}", directory);
}
svn commit: r1840538 - in /jackrabbit/oak/branches/1.6: ./ oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/CIHelper.java oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lu
Author: mduerig Date: Tue Sep 11 10:26:38 2018 New Revision: 1840538 URL: http://svn.apache.org/viewvc?rev=1840538=rev Log: OAK-7708: Backport OAK-6648 to Oak 1.6 Merged r1808053 Modified: jackrabbit/oak/branches/1.6/ (props changed) jackrabbit/oak/branches/1.6/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/CIHelper.java jackrabbit/oak/branches/1.6/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexEditorTest.java Propchange: jackrabbit/oak/branches/1.6/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Sep 11 10:26:38 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808125,1808128,1808142,1808240,1808246,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1837475,1837998,1838637,1839746,1840024 +/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783110,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784034,1784130,1784162,1784251,1784401,1784551,1784574,1784689,1785095,1785108,1785161,1785172,1785283,1785652,1785838,1785916-1785917,1785919,1785946,1786122,1787074,1787145,1787151,1787217,1787425,1788056,1788378,1788387-1788389,1788463,1788476,1788850,1789056,1789534,1789925,1789940,1789987,1790006,1790013,1790069,1790077,1790079,1790382,1790502-1790503,1792049,1792463,1792742,1792746,1793013,1793088,1793618,1793627,1793644,1794393,1794417,1794683,1795138,1795314,1795330,1795475,1795488,1795491,1795502,1795594,1795613,1795618,1796144,1796230,1796239,1796274,1796278,1796988,1797378,1798035,1798832,1798834,1799219,1799389,1799393,1799924,1800244,1800269,1800606,1800613,1800974,1801011,1801013,1801118-1801119 ,1801675,1802260,1802262,1802286,1802548,1802905,1802934,1802938,1802973,1803026,1803247-1803249,1803951,1803953-1803955,1804437,1805851-1805852,1806668,1807308,1807688,1808022,1808053,1808125,1808128,1808142,1808240,1808246,1808731,1809024,1809026,1809131,1809163,1809178-1809179,1809253,1809255-1809256,1809289,1809745,1811071-1811072,1811155,1811380,1811655,1811952,1811963,1811986,1813192,1813538,1814189,1814332,1814397,1814475,1815201,1815438,1815926,1817326,1817919,1817987-1817988,1817990,1818038,1818042,1818056,1818124,1818137,1818554,1818576,1818645,1819048,1819050,1821237,1821325,1821358,1821495,1821516,1821847,1822207,1822850,1823172,1823655,1824896,1825471,1825654,1826237,1826338,1826532,1826640,1826932,1826957,1827472,1827486,1827977,1828502,1829527,1829569,1829587,1829665,1829854,1829864,1829987,1829998,1830019,1830160,1830239,1830748,1831190,1831374,1832379,1832535,1833308,1834648-1834649,1834681,1835060,1837475,1837998,1838637,1839746,1840024 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.6/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/CIHelper.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/CIHelper.java?rev=1840538=1840537=1840538=diff
svn commit: r1837641 - in /jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr: PartialValueFactory.java ValueFactoryImpl.java
Author: mduerig
Date: Wed Aug 8 11:03:44 2018
New Revision: 1837641
URL: http://svn.apache.org/viewvc?rev=1837641=rev
Log:
OAK-7688: Replace usage of static ValueFactoryImpl methods
Consistently use nullability annotations
Modified:
jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/PartialValueFactory.java
jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/ValueFactoryImpl.java
Modified:
jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/PartialValueFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/PartialValueFactory.java?rev=1837641=1837640=1837641=diff
==
---
jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/PartialValueFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/value/jcr/PartialValueFactory.java
Wed Aug 8 11:03:44 2018
@@ -16,6 +16,9 @@
*/
package org.apache.jackrabbit.oak.plugins.value.jcr;
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.jackrabbit.oak.plugins.value.jcr.ValueImpl.newValue;
+
import java.math.BigDecimal;
import java.net.URI;
import java.net.URISyntaxException;
@@ -31,7 +34,6 @@ import javax.jcr.ValueFormatException;
import javax.jcr.nodetype.NodeType;
import com.google.common.collect.Lists;
-
import org.apache.jackrabbit.oak.api.Blob;
import org.apache.jackrabbit.oak.api.PropertyState;
import org.apache.jackrabbit.oak.api.blob.BlobAccessProvider;
@@ -54,9 +56,6 @@ import org.apache.jackrabbit.util.ISO860
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
-import static com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.jackrabbit.oak.plugins.value.jcr.ValueImpl.newValue;
-
/**
* A partial value factory implementation that only deals with in-memory values
* and can wrap a {@link Value} around a {@link PropertyState}.
@@ -66,6 +65,7 @@ public class PartialValueFactory {
/**
* This default blob access provider is a no-op implementation.
*/
+@NotNull
public static final BlobAccessProvider DEFAULT_BLOB_ACCESS_PROVIDER = new
DefaultBlobAccessProvider();
@NotNull
@@ -114,6 +114,7 @@ public class PartialValueFactory {
* @throws IllegalArgumentException if {@code property.isArray()} is
* {@code true}.
*/
+@NotNull
public Value createValue(@NotNull PropertyState property) {
return newValue(property, namePathMapper, blobAccessProvider);
}
@@ -125,6 +126,7 @@ public class PartialValueFactory {
* @param property The property state
* @return A list of new {@code Value} instances
*/
+@NotNull
public List createValues(@NotNull PropertyState property) {
List values = Lists.newArrayList();
for (int i = 0; i < property.count(); i++) {
@@ -135,31 +137,38 @@ public class PartialValueFactory {
//---< ValueFactory
>---
-public Value createValue(String value) {
+@NotNull
+public Value createValue(@NotNull String value) {
return newValue(StringPropertyState.stringProperty("", value),
namePathMapper, getBlobAccessProvider());
}
+@NotNull
public Value createValue(long value) {
return newValue(LongPropertyState.createLongProperty("", value),
namePathMapper, getBlobAccessProvider());
}
+@NotNull
public Value createValue(double value) {
return newValue(DoublePropertyState.doubleProperty("", value),
namePathMapper, getBlobAccessProvider());
}
-public Value createValue(Calendar value) {
+@NotNull
+public Value createValue(@NotNull Calendar value) {
return newValue(PropertyStates.createProperty("", value),
namePathMapper, getBlobAccessProvider());
}
+@NotNull
public Value createValue(boolean value) {
return newValue(BooleanPropertyState.booleanProperty("", value),
namePathMapper, getBlobAccessProvider());
}
-public Value createValue(Node value) throws RepositoryException {
+@NotNull
+public Value createValue(@NotNull Node value) throws RepositoryException {
return createValue(value, false);
}
-public Value createValue(Node value, boolean weak) throws
RepositoryException {
+@NotNull
+public Value createValue(@NotNull Node value, boolean weak) throws
RepositoryException {
if (!value.isNodeType(NodeType.MIX_REFERENCEABLE)) {
throw new ValueFormatException(
"Node is not referenceable: " + val
svn commit: r1836706 - /jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java
Author: mduerig
Date: Thu Jul 26 09:17:56 2018
New Revision: 1836706
URL: http://svn.apache.org/viewvc?rev=1836706=rev
Log:
OAK-7603: [DirectBinaryAccess] Oak API extensions
Typo in Javadoc
Modified:
jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java
Modified:
jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java?rev=1836706=1836705=1836706=diff
==
---
jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java
(original)
+++
jackrabbit/oak/trunk/oak-api/src/main/java/org/apache/jackrabbit/oak/api/blob/BlobAccessProvider.java
Thu Jul 26 09:17:56 2018
@@ -103,7 +103,7 @@ public interface BlobAccessProvider {
throws IllegalArgumentException;
/**
- * Obtain a download URI for a {@link Blob). This is usually a signed URI
+ * Obtain a download URI for a {@link Blob}. This is usually a signed URI
* that can be used to directly download the blob corresponding to the
* provided {@link Blob}.
*
svn commit: r1835059 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Author: mduerig
Date: Wed Jul 4 13:36:32 2018
New Revision: 1835059
URL: http://svn.apache.org/viewvc?rev=1835059=rev
Log:
OAK-7334: Transform CacheWeightEstimator into a unit test
Missing newline in output
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java?rev=1835059=1835058=1835059=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Wed Jul 4 13:36:32 2018
@@ -376,9 +376,9 @@ public class CacheWeightEstimator {
System.out.printf("estimated heap usage: %d bytes. %d bytes per
item\n", heapEstimate, perItemHeapEstimate);
double percentageOff = 100 * ((double) heapEstimate / (double)
heapDelta - 1);
if (percentageOff < 0) {
-System.out.printf("estimated heap usage is %.2f%% to low",
-percentageOff);
+System.out.printf("estimated heap usage is %.2f%% to low\n",
-percentageOff);
} else {
-System.out.printf("estimated heap usage is %.2f%% to high",
percentageOff);
+System.out.printf("estimated heap usage is %.2f%% to high\n",
percentageOff);
}
}
svn commit: r1835058 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Author: mduerig
Date: Wed Jul 4 13:31:39 2018
New Revision: 1835058
URL: http://svn.apache.org/viewvc?rev=1835058=rev
Log:
OAK-7334: Transform CacheWeightEstimator into a unit test
Implementation as unit test and improvements in reporting
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java?rev=1835058=1835057=1835058=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
Wed Jul 4 13:31:39 2018
@@ -18,10 +18,13 @@
*/
package org.apache.jackrabbit.oak.segment;
+import static java.lang.System.getProperty;
import static
org.apache.jackrabbit.oak.segment.Segment.GC_FULL_GENERATION_OFFSET;
import static org.apache.jackrabbit.oak.segment.SegmentCache.newSegmentCache;
import static org.apache.jackrabbit.oak.segment.SegmentVersion.LATEST_VERSION;
+import static org.junit.Assume.assumeTrue;
+import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.AbstractMap.SimpleImmutableEntry;
import java.util.Map.Entry;
@@ -34,18 +37,20 @@ import org.apache.jackrabbit.oak.commons
import org.apache.jackrabbit.oak.segment.CacheWeights.StringCacheWeigher;
import org.apache.jackrabbit.oak.segment.file.PriorityCache;
import org.apache.jackrabbit.oak.segment.memory.MemoryStore;
+import org.junit.Before;
+import org.junit.Test;
/**
* Test/Utility class to measure size in memory for common segment-tar objects.
*
* The test is disabled by default, to run it you need to set the
* {@code CacheWeightsTest} system property:
- * {@code mv clean test -Dtest=CacheWeightsTest -DCacheWeightsTest=true
-Dtest.opts.memory=-Xmx2G}
+ * {@code mvn clean test -Dtest=CacheWeightEstimator -Dtest.opts.memory=-Xmx2G}
*
*
* To collect the results check the
* {@code org.apache.jackrabbit.oak.segment.CacheWeightsTest-output.txt}
file:
- * {@code cat
target/surefire-reports/org.apache.jackrabbit.oak.segment.CacheWeightsTest-output.txt}
+ * {@code cat
target/surefire-reports/org.apache.jackrabbit.oak.segment.CacheWeightEstimator-output.txt}
*
*/
public class CacheWeightEstimator {
@@ -107,32 +112,37 @@ public class CacheWeightEstimator {
Space losses: 0 bytes internal + 0 bytes external = 0 bytes total
*/
+/** Only run if explicitly asked to via -Dtest=SegmentCompactionIT */
+private static final boolean ENABLED =
+
CacheWeightEstimator.class.getSimpleName().equals(getProperty("test"));
-private static MemoryStore store;
+private final MemoryStore store;
-public static void main(String... args) throws Exception {
-run(CacheWeightEstimator::testObjects);
-run(CacheWeightEstimator::testSegmentIds);
-run(CacheWeightEstimator::testSegmentIdsWGc);
-run(CacheWeightEstimator::testRecordIds);
-run(CacheWeightEstimator::testRecordIdsWGc);
-run(CacheWeightEstimator::testStringCache);
-run(CacheWeightEstimator::testNodeCache);
-run(CacheWeightEstimator::testSegments);
-run(CacheWeightEstimator::testSegmentCache);
-run(CacheWeightEstimator::testStrings);
+public CacheWeightEstimator() throws IOException {
+store = new MemoryStore();
}
-private static void run(Runnable runnable) throws Exception {
-store = new MemoryStore();
-try {
-runnable.run();
-} finally {
-store = null;
-}
+public static void main(String... args) throws Exception {
+CacheWeightEstimator cwe = new CacheWeightEstimator();
+cwe.testObjects();
+cwe.testSegmentIds();
+cwe.testSegmentIdsWGc();
+cwe.testRecordIds();
+cwe.testRecordIdsWGc();
+cwe.testStringCache();
+cwe.testNodeCache();
+cwe.testSegments();
+cwe.testSegmentCache();
+cwe.testStrings();
+}
+
+@Before
+public void setup() {
+assumeTrue(ENABLED);
}
-private static void testObjects() {
+@Test
+public void testObjects() {
final int count = 100;
Supplier> factory = () -> {
Object[] objects = new Object[count];
@@ -146,15 +156,16 @@ public class CacheWeightEstimator {
runTest(factory, "Object[x" + count + "]");
}
-private static void testSegmentIds() {
+private void testSegmentIds() {
runSegmentIds(100, false);
}
-p
svn commit: r1834999 - in /jackrabbit/oak/trunk: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ oak-segm
Author: mduerig
Date: Tue Jul 3 16:14:45 2018
New Revision: 1834999
URL: http://svn.apache.org/viewvc?rev=1834999=rev
Log:
OAK-7604: Add FileStoreStatsMBean#getSegmentCount for monitoring the number of
segments
Initial patch
Modified:
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreStats.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreStatsMBean.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/SegmentTarWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarFiles.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/persistence/SegmentArchiveWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/FileStoreTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tar/TarFilesTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java?rev=1834999=1834998=1834999=diff
==
---
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureSegmentArchiveWriter.java
Tue Jul 3 16:14:45 2018
@@ -16,16 +16,8 @@
*/
package org.apache.jackrabbit.oak.segment.azure;
-import com.google.common.base.Stopwatch;
-import com.microsoft.azure.storage.StorageException;
-import com.microsoft.azure.storage.blob.CloudBlobDirectory;
-import com.microsoft.azure.storage.blob.CloudBlockBlob;
-import org.apache.jackrabbit.oak.segment.spi.monitor.FileStoreMonitor;
-import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
-import org.apache.jackrabbit.oak.segment.azure.queue.SegmentWriteAction;
-import org.apache.jackrabbit.oak.segment.azure.queue.SegmentWriteQueue;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveEntry;
-import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveWriter;
+import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.getSegmentFileName;
+import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.readBufferFully;
import java.io.File;
import java.io.IOException;
@@ -38,8 +30,15 @@ import java.util.Optional;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
-import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.getSegmentFileName;
-import static
org.apache.jackrabbit.oak.segment.azure.AzureUtilities.readBufferFully;
+import com.google.common.base.Stopwatch;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.CloudBlobDirectory;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import org.apache.jackrabbit.oak.segment.azure.queue.SegmentWriteAction;
+import org.apache.jackrabbit.oak.segment.azure.queue.SegmentWriteQueue;
+import org.apache.jackrabbit.oak.segment.spi.monitor.FileStoreMonitor;
+import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
+import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentArchiveWriter;
public class AzureSegmentArchiveWriter implements SegmentArchiveWriter {
@@ -150,6 +149,11 @@ public class AzureSegmentArchiveWriter i
}
@Override
+public int getEntryCount() {
+return index.size();
+}
+
+@Override
public void close() throws IOException {
if (queue.isPresent()) { // required to handle IOException
SegmentWriteQueue q = queue.get();
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1834999=1834998=1834999=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Tue Jul 3 16:14:45
svn commit: r1831089 - in /jackrabbit/oak/trunk: oak-doc/src/site/markdown/nodestore/segment/ oak-run/src/main/java/org/apache/jackrabbit/oak/run/ oak-segment-tar/src/main/java/org/apache/jackrabbit/o
Author: mduerig Date: Mon May 7 12:06:14 2018 New Revision: 1831089 URL: http://svn.apache.org/viewvc?rev=1831089=rev Log: OAK-5655: TarMK: Analyse locality of reference Add the capability to analyse random IO traces Added: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/Traces.java jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/RandomAccessTrace.java jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/RandomTraceTest.java Removed: jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Trace.java jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerTest.java Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1831089=1831088=1831089=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Mon May 7 12:06:14 2018 @@ -828,21 +828,27 @@ java -jar oak-run.jar iotrace PATH --tra usage: iotrace path/to/segmentstore Option (* = required) Description - --- ---depth Maximal depth of the traversal (default: 5) +--count Number of paths to access Applies to RANDOM (default: 1000) +--depth Maximal depth of the traversal. Applies to BREADTH, DEPTH (default: 5) --mmapuse memory mapping for the file store (default: true) --output output file where the IO trace is written to (default: iotrace.csv) ---path starting path for the traversal (default: /root) +--path starting path for the traversal. Applies to BREADTH, DEPTH (default: /root) +--paths file containing list of paths to traverse. Applies to RANDOM (default: paths.txt) +--seed Seed for generating random numbers. Applies to RANDOM (default: 0) --segment-cache size of the segment cache in MB (default: 256) ---trace (*) type of the traversal. Either of [DEPTH, BREADTH] - +* --trace type of the traversal. Either of [DEPTH, BREADTH, RANDOM] The `iotrace` command collects IO traces of read accesses to the segment store's back-end (e.g. disk). Traffic patterns can be specified via the `--trace` option. Permissible values -are `DEPTH` for depth first traversal and `BREADTH` for breadth first traversal. The `--depth` -option limits the maximum number of levels traversed. The `--path` option specifies the node -where traversal starts (from the super root). The `--mmap` and `--segment-cache` options -configure memory mapping and segment cache size of the segment store, respectively. The -`--output` options specifies the file where the IO trace is stored. IO traces are stored in +are `DEPTH` for depth first traversal, `BREADTH` for breadth first traversal and `RANDOM` for +random access. The `--depth` option limits the maximum number of levels traversed. +The `--path` option specifies the node where traversal starts (from the super root). +The `--mmap` and `--segment-cache` options configure memory mapping and segment cache size +of the segment store, respectively. +The `--paths` option specifies the list of paths to access. The file must contain a single path +per line. +The `--seed` option specifies the seed to used when randomly choosing a paths. +The `--output` options specifies the file where the IO trace is stored. IO traces are stored in CSV format of the following form: ``` Modified: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java?rev=1831089=1831088=1831089=diff == --- jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java (original) +++ jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java Mon May 7 12:06:14 2018 @@ -19,6 +19,13 @@ package org.apache.jackrabbit.oak.run; import static java.lang.String.format; +import static org.apache.jackrabbit.oak.run.Traces.BREADTH; +import static org.apache.jackrabbit.oak.run.Traces.DEFAULT_COUNT
svn commit: r1830012 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ main/java/org/apache/jackrabbit/oak/segment/file/ main/java/org/apache/jackrabbit/oak/
Author: mduerig
Date: Tue Apr 24 15:37:15 2018
New Revision: 1830012
URL: http://svn.apache.org/viewvc?rev=1830012=rev
Log:
OAK-7444: Allow collection of IOTraces during normal operation
* Refactoring of IOTracer and related classes to enable logging IO reads via
IOMonitor
* Use logger org.apache.jackrabbit.oak.segment.file.tar.SegmentTarWriter for
logging IO reads in debug mode
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/spi/monitor/CompositeIOMonitor.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DefaultIOTraceWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTraceLogWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTraceMonitor.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTraceWriter.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/spi/
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/spi/monitor/
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/spi/monitor/CompositeIOMonitorTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTraceMonitorTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreBuilder.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java?rev=1830012=1830011=1830012=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
Tue Apr 24 15:37:15 2018
@@ -119,6 +119,7 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.file.FileStoreStatsMBean;
import org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException;
import org.apache.jackrabbit.oak.segment.file.MetricsIOMonitor;
+import org.apache.jackrabbit.oak.segment.file.tar.SegmentTarReader;
import
org.apache.jackrabbit.oak.segment.spi.persistence.SegmentNodeStorePersistence;
import org.apache.jackrabbit.oak.segment.util.RoleUtils;
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
@@ -486,6 +487,7 @@ public class SegmentNodeStoreService {
.withMemoryMapping(configuration.getMemoryMapping())
.withGCMonitor(gcMonitor)
.withIOMonitor(new MetricsIOMonitor(statisticsProvider))
+.withIOLogging(LoggerFactory.getLogger(SegmentTarReader.class))
.withStatisticsProvider(statisticsProvider)
.withGCOptions(gcOptions);
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreBuilder.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreBuilder.java?rev=1830012=1830011=1830012=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreBuilder.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreBuilder.java
Tue Apr 24 15:37:15 2018
@@ -21,6 +21,7 @@ package org.apache.jackrabbit.oak.segmen
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
+import static com.google.common.collect.Sets.newHashSet;
import static
org.apache.jackrabbit.oak.segment.CachingSegmentReader.DEFAULT_STRING_CACHE_MB;
import static
org.apache.jackrabbit.oak.segment.CachingSegmentReader.DEFAULT_TEMPLATE_CACHE_MB;
import static
org.apache.jackrabbit.oak.segment.SegmentCache.DEFAULT_SEGMENT_CACHE_MB;
@@ -32,6 +33,7 @@ import static org.apache.jackrabbit.oak.
import java.io.File;
import java.io.IOException;
+import java.util.Set;
import javax.annotation.CheckForNull;
import javax.annotation.Nonnull;
@@ -47,9
svn commit: r1829969 - in /jackrabbit/oak/trunk: oak-doc/src/site/markdown/nodestore/segment/ oak-run/ oak-run/src/main/java/org/apache/jackrabbit/oak/run/ oak-segment-tar/src/main/java/org/apache/jac
Author: mduerig Date: Tue Apr 24 08:56:59 2018 New Revision: 1829969 URL: http://svn.apache.org/viewvc?rev=1829969=rev Log: OAK-7402: Expose UI for collecting IO traces Introduce a iotrace run mode in oak-run Added: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java Removed: jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md jackrabbit/oak/trunk/oak-run/README.md jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1829969=1829968=1829969=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Tue Apr 24 08:56:59 2018 @@ -30,6 +30,7 @@ * [Check](#check) * [Compact](#compact) * [Debug](#debug) +* [IOTrace](#iotrace) * [Diff](#diff) * [History](#history) @@ -819,6 +820,38 @@ The pair of record IDs can be followed b When a node record ID range is specified, the tool will perform a diff between the two nodes pointed by the record IDs, optionally following the provided path. The result of the diff will be printed in JSOP format. +### IOTrace + + +java -jar oak-run.jar iotrace PATH --trace DEPTH|BREADTH [--depth DEPTH] [--mmap MMAP] [--output OUTPUT] [--path PATH] [--segment-cache SEGMENT_CACHE] + +usage: iotrace path/to/segmentstore +Option (* = required) Description +- --- +--depth Maximal depth of the traversal (default: 5) +--mmapuse memory mapping for the file store (default: true) +--output output file where the IO trace is written to (default: iotrace.csv) +--path starting path for the traversal (default: /root) +--segment-cache size of the segment cache in MB (default: 256) +--trace (*) type of the traversal. Either of [DEPTH, BREADTH] + + +The `iotrace` command collects IO traces of read accesses to the segment store's back-end +(e.g. disk). Traffic patterns can be specified via the `--trace` option. Permissible values +are `DEPTH` for depth first traversal and `BREADTH` for breadth first traversal. The `--depth` +option limits the maximum number of levels traversed. The `--path` option specifies the node +where traversal starts (from the super root). The `--mmap` and `--segment-cache` options +configure memory mapping and segment cache size of the segment store, respectively. The +`--output` options specifies the file where the IO trace is stored. IO traces are stored in +CSV format of the following form: + +``` +timestamp,file,segmentId,length,elapsed +1522147945084,data01415a.tar,f81378df-b3f8-4b25--0002c450,181328,171849 +1522147945096,data01415a.tar,f81378df-b3f8-4b25--0002c450,181328,131272 +1522147945097,data01415a.tar,f81378df-b3f8-4b25--0002c450,181328,142766 +``` + ### Diff ``` Modified: jackrabbit/oak/trunk/oak-run/README.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/README.md?rev=1829969=1829968=1829969=diff == --- jackrabbit/oak/trunk/oak-run/README.md (original) +++ jackrabbit/oak/trunk/oak-run/README.md Tue Apr 24 08:56:59 2018 @@ -27,6 +27,7 @@ The following runmodes are currently ava * unlockUpgrade : Unlock a DocumentMK upgrade to a newer version * upgrade : Migrate existing Jackrabbit 2.x repository to Oak. * export : Export repository content as json +* iotrace : Collect a trace of segment store read accesses Some of the features related to Jackrabbit 2.x are provided by oak-run-jr2 jar. See @@ -78,6 +79,12 @@ Debug See the [official documentation](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html#debug). +IOTrace +- + +See the [official documentation](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html#iotrace). + + Console --- Modified: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java?rev=1829969=1829968=1829969=diff == --- jackrabbit/oak/trunk/oak-run/src/main/java/org
svn commit: r1828873 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
Author: mduerig
Date: Wed Apr 11 08:07:19 2018
New Revision: 1828873
URL: http://svn.apache.org/viewvc?rev=1828873=rev
Log:
OAK-7402: Expose UI for collecting IO traces
Updated issue reference in FIXME
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java?rev=1828873=1828872=1828873=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
Wed Apr 11 08:07:19 2018
@@ -58,7 +58,7 @@ import org.junit.Test;
* {@code -Ddepth=n}. Default {@code 5}
* {@code -Dpath=/path/to/start/node}. Default {@code /root}
*
- * FIXME OAK-5655: Turn this into a development tool and move to the right
place.
+ * FIXME OAK-7402: Turn this into a development tool and move to the right
place.
*/
public class IOTracerRunner extends IOMonitorAdapter {
private static final boolean ENABLED =
svn commit: r1828740 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java
Author: mduerig
Date: Mon Apr 9 15:39:09 2018
New Revision: 1828740
URL: http://svn.apache.org/viewvc?rev=1828740=rev
Log:
OAK-7396: ReadOnlyFileStore.readSegment does not correctly throw
SegmentNotFoundException
Catch UncheckedExecutionException along with ExecutionException
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java?rev=1828740=1828739=1828740=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ReadOnlyFileStore.java
Mon Apr 9 15:39:09 2018
@@ -32,6 +32,7 @@ import java.util.concurrent.ExecutionExc
import javax.annotation.Nonnull;
import com.google.common.io.Closer;
+import com.google.common.util.concurrent.UncheckedExecutionException;
import org.apache.jackrabbit.oak.segment.RecordId;
import org.apache.jackrabbit.oak.segment.Segment;
import org.apache.jackrabbit.oak.segment.SegmentId;
@@ -117,7 +118,7 @@ public class ReadOnlyFileStore extends A
return readSegmentUncached(tarFiles, id);
}
});
-} catch (ExecutionException e) {
+} catch (ExecutionException | UncheckedExecutionException e) {
throw asSegmentNotFoundException(e, id);
}
}
svn commit: r1828244 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/ test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/
Author: mduerig
Date: Tue Apr 3 14:50:24 2018
New Revision: 1828244
URL: http://svn.apache.org/viewvc?rev=1828244=rev
Log:
OAK-5655: TarMK: Analyse locality of reference
Depth first trace
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTrace.java
- copied, changed from r1828237,
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTraceTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java?rev=1828244=1828243=1828244=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
Tue Apr 3 14:50:24 2018
@@ -40,7 +40,7 @@ import org.apache.jackrabbit.oak.spi.sta
* A breadth first traversal trace.
*
* When {@link Trace#run(NodeState) run} this trace performs a breadth first
traversal starting
- * from the passed node up to a certain depth. It logs the current depth and
the number of
+ * from the passed node down to a certain depth. It logs the current depth and
the number of
* traversed nodes as additional {@link IOTracer#setContext(List) context}.
*/
public class BreadthFirstTrace implements Trace {
@@ -92,7 +92,7 @@ public class BreadthFirstTrace implement
return node;
}
-private void traverse(Queue nodes, int depth) {
+private void traverse(@Nonnull Queue nodes, int depth) {
if (!nodes.isEmpty()) {
Queue children = newLinkedList();
while (!nodes.isEmpty()) {
Copied:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTrace.java
(from r1828237,
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java)
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTrace.java?p2=jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTrace.java=jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java=1828237=1828244=1828244=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/DepthFirstTrace.java
Tue Apr 3 14:50:24 2018
@@ -19,14 +19,12 @@
package org.apache.jackrabbit.oak.segment.tool.iotrace;
import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.collect.Lists.newLinkedList;
import static java.lang.String.valueOf;
-import static java.util.Collections.singleton;
+import static org.apache.jackrabbit.oak.commons.PathUtils.concat;
import static org.apache.jackrabbit.oak.commons.PathUtils.elements;
import java.io.Writer;
import java.util.List;
-import java.util.Queue;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
import java.util.function.Function;
@@ -37,20 +35,20 @@ import com.google.common.collect.Immutab
import org.apache.jackrabbit.oak.spi.state.NodeState;
/**
- * A breadth first traversal trace.
+ * A depth first traversal trace.
*
- * When {@link Trace#run(NodeState) run} this trace performs a breadth first
traversal starting
- * from the passed node up to a certain depth. It logs the current depth and
the number of
- * traversed nodes as additional {@link IOTracer#setContext(List) context}.
+ * When {@link Trace#run(NodeState) run} this trace performs a depth first
traversal starting
+ * from the passed node down to a certain depth. It logs the current depth,
the number of traversed
+ * nodes and the current path as additional {@link IOTracer#setContext(List)
context}.
*/
-public class
svn commit: r1828237 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/ test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/
Author: mduerig
Date: Tue Apr 3 13:38:17 2018
New Revision: 1828237
URL: http://svn.apache.org/viewvc?rev=1828237=rev
Log:
OAK-5655: TarMK: Analyse locality of reference
Typo
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
- copied, changed from r1827987,
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTraceTest.java
- copied, changed from r1827987,
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java
Removed:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerTest.java
Copied:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
(from r1827987,
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java)
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java?p2=jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java=jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java=1827987=1828237=1828237=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTrace.java
Tue Apr 3 13:38:17 2018
@@ -37,13 +37,13 @@ import com.google.common.collect.Immutab
import org.apache.jackrabbit.oak.spi.state.NodeState;
/**
- * A breath first traversal trace.
+ * A breadth first traversal trace.
*
- * When {@link Trace#run(NodeState) run} this trace performs a breath first
traversal starting
+ * When {@link Trace#run(NodeState) run} this trace performs a breadth first
traversal starting
* from the passed node up to a certain depth. It logs the current depth and
the number of
* traversed nodes as additional {@link IOTracer#setContext(List) context}.
*/
-public class BreathFirstTrace implements Trace {
+public class BreadthFirstTrace implements Trace {
/**
* The context specification of this trace.
@@ -64,12 +64,12 @@ public class BreathFirstTrace implements
private final AtomicInteger nodeCount = new AtomicInteger();
/**
- * Create a new instance of a breath first traversal trace.
+ * Create a new instance of a breadth first traversal trace.
* @param depth maximal depth of the nodes to traverse
* @param path path of the root node where to start traversing
* @param context consumer to pass the additional context to
*/
-public BreathFirstTrace(int depth, @Nonnull String path, @Nonnull
Consumer<List> context) {
+public BreadthFirstTrace(int depth, @Nonnull String path, @Nonnull
Consumer<List> context) {
checkArgument(depth >= 0);
this.depth = depth;
Copied:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTraceTest.java
(from r1827987,
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java)
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTraceTest.java?p2=jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTraceTest.java=jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java=1827987=1828237=1828237=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreadthFirstTraceTest.java
Tue Apr 3 13:38:17 2018
@@ -31,7 +31,7 @@ import org.apache.jackrabbit.oak.
svn commit: r1827901 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
Author: mduerig
Date: Wed Mar 28 08:05:14 2018
New Revision: 1827901
URL: http://svn.apache.org/viewvc?rev=1827901=rev
Log:
OAK-5655: TarMK: Analyse locality of reference
Fixed typo causing bad UUIDs in io traces
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java?rev=1827901=1827900=1827901=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
Wed Mar 28 08:05:14 2018
@@ -197,7 +197,7 @@ public class IOTracer {
@Override
public String toString() {
return System.currentTimeMillis() + "," + fileName + "," +
-new UUID(msb, length) + "," + length + "," + elapsed +
"," +
+new UUID(msb, lsb) + "," + length + "," + elapsed +
"," +
join(",", context);
}
}
svn commit: r1827841 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/ test/java/org/apache/jackrabbit/oak/segment/tool/ test/java/org/apache/ja
Author: mduerig
Date: Tue Mar 27 14:34:31 2018
New Revision: 1827841
URL: http://svn.apache.org/viewvc?rev=1827841=rev
Log:
OAK-5655: TarMK: Analyse locality of reference
Utility for collecting IO traces for given read access patterns
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracer.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Trace.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTraceTest.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerTest.java
Added:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java?rev=1827841=auto
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
(added)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/BreathFirstTrace.java
Tue Mar 27 14:34:31 2018
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.jackrabbit.oak.segment.tool.iotrace;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.Lists.newLinkedList;
+import static java.lang.String.valueOf;
+import static java.util.Collections.singleton;
+import static org.apache.jackrabbit.oak.commons.PathUtils.elements;
+
+import java.io.Writer;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+
+import javax.annotation.Nonnull;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.jackrabbit.oak.spi.state.NodeState;
+
+/**
+ * A breath first traversal trace.
+ *
+ * When {@link Trace#run(NodeState) run} this trace performs a breath first
traversal starting
+ * from the passed node up to a certain depth. It logs the current depth and
the number of
+ * traversed nodes as additional {@link IOTracer#setContext(List) context}.
+ */
+public class BreathFirstTrace implements Trace {
+
+/**
+ * The context specification of this trace.
+ * @see IOTracer#newIOTracer(Function, Writer, String)
+ */
+@Nonnull
+public static final String CONTEXT_SPEC = "depth,count";
+
+private final int depth;
+
+@Nonnull
+private final String path;
+
+@Nonnull
+private final Consumer<List> context;
+
+@Nonnull
+private final AtomicInteger nodeCount = new AtomicInteger();
+
+/**
+ * Create a new instance of a breath first traversal trace.
+ * @param depth maximal depth of the nodes to traverse
+ * @param path path of the root node where to start traversing
+ * @param context consumer to pass the additional context to
+ */
+public BreathFirstTrace(int depth, @Nonnull String path, @Nonnull
Consumer<List> context) {
+checkArgument(depth >= 0);
+
+this.depth = depth;
+this.path = path;
+this.context = context;
+}
+
+@Override
+public void run(@Nonnull NodeState node) {
+updateContext(context, 0, nodeCount.incrementAndGet());
+traverse(new
svn commit: r1826729 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
Author: mduerig
Date: Wed Mar 14 15:14:20 2018
New Revision: 1826729
URL: http://svn.apache.org/viewvc?rev=1826729=rev
Log:
OAK-7326: Add a way to disable the SegmentCache
Immediately evict segments on put
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java?rev=1826729=1826728=1826729=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
Wed Mar 14 15:14:20 2018
@@ -239,7 +239,9 @@ public abstract class SegmentCache {
}
@Override
-public void putSegment(@Nonnull Segment segment) {}
+public void putSegment(@Nonnull Segment segment) {
+segment.getSegmentId().unloaded();
+}
@Override
public void clear() {}
svn commit: r1826359 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/ main/java/org/apache/jackrabbit/oak/segment/file/ test/java/org/apache/jackrabbit/oak/
Author: mduerig
Date: Fri Mar 9 16:45:34 2018
New Revision: 1826359
URL: http://svn.apache.org/viewvc?rev=1826359=rev
Log:
OAK-7326: Add a way to disable the SegmentCache
Abstract the SegmentCache and return an always empty cache when size <= 0
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentId.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractFileStore.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CacheWeightEstimator.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCacheTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java?rev=1826359=1826358=1826359=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java
Fri Mar 9 16:45:34 2018
@@ -19,12 +19,14 @@
package org.apache.jackrabbit.oak.segment;
+import static com.google.common.base.Preconditions.checkNotNull;
import static org.apache.jackrabbit.oak.segment.CacheWeights.segmentWeight;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
import javax.annotation.Nonnull;
@@ -45,56 +47,27 @@ import org.apache.jackrabbit.oak.segment
* SegmentId#segment}. Every time an segment is evicted from this cache the
* memoised segment is discarded (see {@link SegmentId#onAccess}.
*/
-public class SegmentCache {
+public abstract class SegmentCache {
/**
* Default maximum weight of this cache in MB
*/
public static final int DEFAULT_SEGMENT_CACHE_MB = 256;
-/**
- * Maximum weight of the items in this cache
- */
-private final long maximumWeight;
-
-/**
- * Cache of recently accessed segments
- */
-@Nonnull
-private final Cache<SegmentId, Segment> cache;
+private static final String NAME = "Segment Cache";
/**
- * Statistics of this cache. Do to the special access patter (see class
- * comment), we cannot rely on {@link Cache#stats()}.
- */
-@Nonnull
-private final Stats stats = new Stats("Segment Cache");
-
-/**
- * Create a new segment cache of the given size.
+ * Create a new segment cache of the given size. Returns an always empty
+ * cache for {@code cacheSizeMB <= 0}.
*
* @param cacheSizeMB size of the cache in megabytes.
*/
-public SegmentCache(long cacheSizeMB) {
-this.maximumWeight = cacheSizeMB * 1024 * 1024;
-this.cache = CacheBuilder.newBuilder()
-.concurrencyLevel(16)
-.maximumWeight(maximumWeight)
-.weigher(new SegmentCacheWeigher())
-.removalListener(this::onRemove)
-.build();
-}
-
-/**
- * Removal handler called whenever an item is evicted from the cache.
- */
-private void onRemove(@Nonnull RemovalNotification<SegmentId, Segment>
notification) {
-stats.evictionCount.incrementAndGet();
-if (notification.getValue() != null) {
-
stats.currentWeight.addAndGet(-segmentWeight(notification.getValue()));
-}
-if (notification.getKey() != null) {
-notification.getKey().unloaded();
+@Nonnull
+public static SegmentCache newSegmentCache(long cacheSizeMB) {
+if (cacheSizeMB > 0) {
+return new NonEmptyCache(cacheSizeMB);
+} else {
+return new EmptyCache();
}
}
@@ -108,31 +81,8 @@ public class SegmentCache {
* @throws ExecutionException when {@code loader} failed to load an segment
*/
@Nonnull
-public Segment getSegment(@Nonnull final SegmentId id, @Nonnull final
Callable loader) throws ExecutionException {
-if (id.isDataSegmentId()) {
-return cache.get(id, () -> {
-try {
-long t0 = System.nanoTime();
-Segment segment = loader.call();
-stats.loadSuccessCount.incrementAndGet();
-stats.loadTime.addAndGet(System.nanoTime() - t0);
-stats.missCount.incrementAndGet();
-stats.currentWeight.addAndGet(segmentWeight(segment));
-id.
svn commit: r1826132 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Author: mduerig
Date: Wed Mar 7 16:18:23 2018
New Revision: 1826132
URL: http://svn.apache.org/viewvc?rev=1826132=rev
Log:
OAK-7317: SegmentParser#parseBlob does not long ids of external blobs
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java?rev=1826132=1826131=1826132=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentParser.java
Wed Mar 7 16:18:23 2018
@@ -19,17 +19,20 @@
package org.apache.jackrabbit.oak.segment;
+import static com.google.common.base.Charsets.UTF_8;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.collect.Lists.newArrayList;
import static com.google.common.collect.Lists.newArrayListWithCapacity;
import static java.util.Collections.singletonList;
+import static java.util.Objects.requireNonNull;
import static org.apache.jackrabbit.oak.api.Type.BINARY;
-import static org.apache.jackrabbit.oak.segment.SegmentStream.BLOCK_SIZE;
import static org.apache.jackrabbit.oak.segment.ListRecord.LEVEL_SIZE;
import static org.apache.jackrabbit.oak.segment.Segment.MEDIUM_LIMIT;
import static org.apache.jackrabbit.oak.segment.Segment.RECORD_ID_BYTES;
import static org.apache.jackrabbit.oak.segment.Segment.SMALL_LIMIT;
+import static org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId;
+import static org.apache.jackrabbit.oak.segment.SegmentStream.BLOCK_SIZE;
import static org.apache.jackrabbit.oak.segment.Template.MANY_CHILD_NODES;
import static org.apache.jackrabbit.oak.segment.Template.ZERO_CHILD_NODES;
@@ -674,8 +677,13 @@ public class SegmentParser {
size += (8 + RECORD_ID_BYTES + length);
blobType = BlobType.LONG;
} else if ((head & 0xf0) == 0xe0) {
-// 1110 : external value
-int length = (head & 0x0f) << 8 |
(segment.readByte(blobId.getRecordNumber(), 1) & 0xff);
+// 1110 : external value, short blob ID
+int length = UTF_8.encode(requireNonNull(readBlobId(segment,
blobId.getRecordNumber(.limit();
+size += (2 + length);
+blobType = BlobType.EXTERNAL;
+} else if ((head & 0xf8) == 0xf0) {
+// 0xxx: external value, long blob ID
+int length = UTF_8.encode(requireNonNull(readBlobId(segment,
blobId.getRecordNumber(.limit();
size += (2 + length);
blobType = BlobType.EXTERNAL;
} else {
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java?rev=1826132=1826131=1826132=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Wed Mar 7 16:18:23 2018
@@ -64,7 +64,6 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.memory.MemoryStore;
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
-import org.junit.Assume;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -374,8 +373,6 @@ public class SegmentParserTest {
@Test
public void longBlob() throws IOException {
SegmentBlob blob = new SegmentBlob(store.getBlobStore(),
writer.writeBlob(createRandomBlob(MEDIUM_LIMIT)));
-// FIXME OAK-7317: SegmentParser#parseBlob does not long ids of
external blobs
-Assume.assumeTrue(blob.getBlobId() == null ||
"shortId".equals(blob.getBlobId()));
ValueInfo valueInfo = new TestParser(store.getReader(), "longBlob") {
@Override
protected void onBlob(RecordId parentId, RecordId blobId) {
svn commit: r1826123 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Author: mduerig
Date: Wed Mar 7 15:55:12 2018
New Revision: 1826123
URL: http://svn.apache.org/viewvc?rev=1826123=rev
Log:
OAK-7317: SegmentParser#parseBlob does not long ids of external blobs
Added ignored test case
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java?rev=1826123=1826122=1826123=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentParserTest.java
Wed Mar 7 15:55:12 2018
@@ -20,6 +20,7 @@
package org.apache.jackrabbit.oak.segment;
import static com.google.common.base.Strings.repeat;
+import static com.google.common.collect.Lists.newArrayList;
import static com.google.common.collect.Lists.newArrayListWithCapacity;
import static com.google.common.collect.Maps.newHashMap;
import static org.apache.jackrabbit.oak.api.Type.BINARY;
@@ -30,6 +31,7 @@ import static org.apache.jackrabbit.oak.
import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static org.apache.jackrabbit.oak.segment.Segment.MEDIUM_LIMIT;
import static org.apache.jackrabbit.oak.segment.Segment.SMALL_LIMIT;
+import static
org.apache.jackrabbit.oak.segment.SegmentParser.BlobType.EXTERNAL;
import static org.apache.jackrabbit.oak.segment.SegmentParser.BlobType.LONG;
import static org.apache.jackrabbit.oak.segment.SegmentParser.BlobType.MEDIUM;
import static org.apache.jackrabbit.oak.segment.SegmentParser.BlobType.SMALL;
@@ -37,12 +39,19 @@ import static org.apache.jackrabbit.oak.
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
import java.io.IOException;
+import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Random;
+import javax.annotation.CheckForNull;
+
+import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import org.apache.jackrabbit.oak.api.Blob;
import org.apache.jackrabbit.oak.api.Type;
@@ -53,12 +62,37 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.SegmentParser.NodeInfo;
import org.apache.jackrabbit.oak.segment.SegmentParser.ValueInfo;
import org.apache.jackrabbit.oak.segment.memory.MemoryStore;
+import org.apache.jackrabbit.oak.spi.blob.BlobStore;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
+import org.junit.Assume;
import org.junit.Before;
import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+@RunWith(Parameterized.class)
public class SegmentParserTest {
+@Parameterized.Parameters(name="{1}")
+public static Collection<Object[]> fixtures() throws Exception {
+BlobStore shortIdBlobStore = mock(BlobStore.class);
+when(shortIdBlobStore.writeBlob(any())).thenReturn("shortId");
+
+BlobStore longIdBlobStore = mock(BlobStore.class);
+
when(longIdBlobStore.writeBlob(any())).thenReturn(Strings.repeat("shortId",
1000));
+
+return newArrayList(
+new Object[]{null, "No BlobStore"},
+new Object[]{shortIdBlobStore, "Short Id BlobStore"},
+new Object[]{longIdBlobStore, "Long Id BlobStore"});
+}
+
+private final BlobStore blobStore;
+
+public SegmentParserTest(BlobStore blobStore, String name) {
+this.blobStore = blobStore;
+}
+
private MemoryStore store;
private DefaultSegmentWriter writer;
@@ -138,7 +172,13 @@ public class SegmentParserTest {
@Before
public void setup() throws IOException {
-store = new MemoryStore();
+store = new MemoryStore() {
+@CheckForNull
+@Override
+public BlobStore getBlobStore() {
+return blobStore;
+}
+};
writer = defaultSegmentWriterBuilder("").build(store);
}
@@ -334,12 +374,14 @@ public class SegmentParserTest {
@Test
public void longBlob() throws IOException {
SegmentBlob blob = new SegmentBlob(store.getBlobStore(),
writer.writeBlob(createRandomBlob(MEDIUM_LIMIT)));
+// FIXME OAK-7317: SegmentParser#parseBlob does not long ids of
external blobs
+Assume.assumeTrue(blob.getBlobId() == null ||
"
svn commit: r1826121 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java
Author: mduerig
Date: Wed Mar 7 15:05:40 2018
New Revision: 1826121
URL: http://svn.apache.org/viewvc?rev=1826121=rev
Log:
@trivial: correctly sort modifiers
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java?rev=1826121=1826120=1826121=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/BlobIdRecordTest.java
Wed Mar 7 15:05:40 2018
@@ -42,7 +42,7 @@ import org.junit.rules.TemporaryFolder;
public class BlobIdRecordTest {
-private static abstract class IdMappingBlobStore implements BlobStore {
+private abstract static class IdMappingBlobStore implements BlobStore {
private final MemoryBlobStore bs = new MemoryBlobStore();
svn commit: r1825650 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
Author: mduerig
Date: Thu Mar 1 14:27:37 2018
New Revision: 1825650
URL: http://svn.apache.org/viewvc?rev=1825650=rev
Log:
OAK-7057: Segment.toString: Record table should include an index into the
hexdump
List both, record offset and record address in the table of records
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java?rev=1825650=1825649=1825650=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
Thu Mar 1 14:27:37 2018
@@ -562,7 +562,10 @@ public class Segment {
writer.format("reference %02x: %s%n", i++, segmentId);
}
for (Entry entry : recordNumbers) {
-writer.format("%10s record %08x: %08x%n", entry.getType(),
entry.getRecordNumber(), entry.getOffset());
+int offset = entry.getOffset();
+int address = data.size() - (MAX_SEGMENT_SIZE - offset);
+writer.format("%10s record %08x: %08x @ %08x%n",
+ entry.getType(), entry.getRecordNumber(),
offset, address);
}
}
writer.println("--");
svn commit: r1825647 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Author: mduerig
Date: Thu Mar 1 14:10:19 2018
New Revision: 1825647
URL: http://svn.apache.org/viewvc?rev=1825647=rev
Log:
OAK-6891: Executions of background threads might pile up
Switch flush thread, filer reaper and disk space check to
Scheduler.scheduleWithFixedDelay
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1825647=1825646=1825647=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Thu Mar 1 14:10:19 2018
@@ -191,13 +191,13 @@ public class FileStore extends AbstractF
this.snfeListener = builder.getSnfeListener();
-fileStoreScheduler.scheduleAtFixedRate(format("TarMK flush [%s]",
directory), 5, SECONDS,
- this::tryFlush);
+fileStoreScheduler.scheduleWithFixedDelay(format("TarMK flush [%s]",
directory), 5, SECONDS,
+ this::tryFlush);
-fileStoreScheduler.scheduleAtFixedRate(format("TarMK filer reaper
[%s]", directory), 5, SECONDS,
- fileReaper::reap);
+fileStoreScheduler.scheduleWithFixedDelay(format("TarMK filer reaper
[%s]", directory), 5, SECONDS,
+ fileReaper::reap);
-fileStoreScheduler.scheduleAtFixedRate(format("TarMK disk space check
[%s]", directory), 1, MINUTES, () -> {
+fileStoreScheduler.scheduleWithFixedDelay(format("TarMK disk space
check [%s]", directory), 1, MINUTES, () -> {
try (ShutDownCloser ignore = shutDown.tryKeepAlive()) {
if (shutDown.isShutDown()) {
log.debug("Shut down in progress, skipping disk space
check");
svn commit: r1825644 - in /jackrabbit/oak/trunk/oak-segment-tar/src: main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java
Author: mduerig
Date: Thu Mar 1 13:23:54 2018
New Revision: 1825644
URL: http://svn.apache.org/viewvc?rev=1825644=rev
Log:
OAK-6891: Executions of background threads might pile up
Added Scheduler.scheduleWithFixedDelay
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java?rev=1825644=1825643=1825644=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Scheduler.java
Thu Mar 1 13:23:54 2018
@@ -123,6 +123,23 @@ public class Scheduler implements Closea
}
/**
+ * Run {@code task} regularly after a fixed delay. The background thread's
name is
+ * set to {@code name} during execution of {@code task}.
+ * @param name
+ * @param delay
+ * @param unit
+ * @param task
+ * @see ScheduledExecutorService#scheduleWithFixedDelay(Runnable, long,
long, TimeUnit)
+ */
+public void scheduleWithFixedDelay(
+@Nonnull String name,
+long delay,
+@Nonnull TimeUnit unit,
+@Nonnull Runnable task) {
+executor.scheduleWithFixedDelay(new SafeRunnable(name, task), delay,
delay, unit);
+}
+
+/**
* Close this scheduler.
* @see ScheduledExecutorService#shutdown()
*/
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java?rev=1825644=1825643=1825644=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/SchedulerTest.java
Thu Mar 1 13:23:54 2018
@@ -69,6 +69,17 @@ public class SchedulerTest {
assertEquals("scheduleAtFixedRate", task.getThreadName());
}
+@Test
+public void scheduleWithFixedDelay() throws Exception {
+TestTask task = new TestTask(5);
+scheduler.scheduleWithFixedDelay("scheduleWithFixedDelay", 200,
MILLISECONDS, task);
+
+assertNull(task.getThreadName());
+assertTrue(task.await());
+assertEquals("scheduleWithFixedDelay", task.getThreadName());
+}
+
+
private static class TestTask implements Runnable {
private final AtomicReference threadName = new
AtomicReference<>();
private final CountDownLatch done;
svn commit: r1825636 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml
Author: mduerig Date: Thu Mar 1 10:01:12 2018 New Revision: 1825636 URL: http://svn.apache.org/viewvc?rev=1825636=rev Log: OAK-7298: Remove debug logging to the console during tests Modified: jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml Modified: jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml?rev=1825636=1825635=1825636=diff == --- jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml (original) +++ jackrabbit/oak/trunk/oak-segment-tar/src/test/resources/logback-test.xml Thu Mar 1 10:01:12 2018 @@ -21,10 +21,6 @@ - - - -
svn commit: r1825635 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Author: mduerig
Date: Thu Mar 1 09:50:39 2018
New Revision: 1825635
URL: http://svn.apache.org/viewvc?rev=1825635=rev
Log:
@trivial: missing @Override
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java?rev=1825635=1825634=1825635=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java
Thu Mar 1 09:50:39 2018
@@ -178,6 +178,7 @@ public class DefaultSegmentWriter implem
});
}
+@Override
@Nonnull
public RecordId writeBlob(@Nonnull final Blob blob) throws IOException {
return writeOperationHandler.execute(new SegmentWriteOperation() {
svn commit: r1825585 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
Author: mduerig
Date: Wed Feb 28 17:03:10 2018
New Revision: 1825585
URL: http://svn.apache.org/viewvc?rev=1825585=rev
Log:
OAK-7058: oak-run compact reports success even when it was cancelled
Print a message and return 1 on cancellation
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java?rev=1825585=1825584=1825585=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/Compact.java
Wed Feb 28 17:03:10 2018
@@ -31,10 +31,10 @@ import static org.apache.jackrabbit.oak.
import java.io.File;
import java.io.IOException;
import java.io.PrintStream;
-import java.io.RandomAccessFile;
import java.util.Date;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+
import javax.annotation.Nullable;
import com.google.common.base.Stopwatch;
@@ -272,7 +272,10 @@ public class Compact {
Stopwatch watch = Stopwatch.createStarted();
try (FileStore store = newFileStore()) {
-store.compactFull();
+if (!store.compactFull()) {
+System.out.printf("Compaction cancelled after %s.\n",
printableStopwatch(watch));
+return 1;
+}
System.out.printf("-> cleaning up\n");
store.cleanup();
JournalFile journal = new LocalJournalFile(path, "journal.log");
@@ -289,7 +292,7 @@ public class Compact {
} catch (Exception e) {
watch.stop();
e.printStackTrace(System.err);
-System.out.printf("Compaction failed in %s.\n",
printableStopwatch(watch));
+System.out.printf("Compaction failed after %s.\n",
printableStopwatch(watch));
return 1;
}
svn commit: r1825582 - /jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
Author: mduerig
Date: Wed Feb 28 16:40:06 2018
New Revision: 1825582
URL: http://svn.apache.org/viewvc?rev=1825582=rev
Log:
OAK-6707: TarWriter.close() must not throw an exception on subsequent
invocations
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java?rev=1825582=1825581=1825582=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarWriter.java
Wed Feb 28 16:40:06 2018
@@ -201,7 +201,9 @@ class TarWriter implements Closeable {
// this part, as no other synchronized methods should get invoked
// once close() has been initiated (see related checkState calls).
synchronized (this) {
-checkState(!closed);
+if (closed) {
+return;
+}
closed = true;
}
svn commit: r1823393 - /jackrabbit/oak/trunk/oak-run/README.md
Author: mduerig Date: Tue Feb 6 21:01:00 2018 New Revision: 1823393 URL: http://svn.apache.org/viewvc?rev=1823393=rev Log: OAK-4146: Improve tarmkrecovery docs Removing obsolete documentation Modified: jackrabbit/oak/trunk/oak-run/README.md Modified: jackrabbit/oak/trunk/oak-run/README.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/README.md?rev=1823393=1823392=1823393=diff == --- jackrabbit/oak/trunk/oak-run/README.md (original) +++ jackrabbit/oak/trunk/oak-run/README.md Tue Feb 6 21:01:00 2018 @@ -23,7 +23,6 @@ The following runmodes are currently ava * revisions : Revision GC on a DocumentMK * server : Run the Oak Server. * tarmkdiff : Show changes between revisions on TarMk -* tarmkrecovery : Lists candidates for head journal entries * tika: Performs text extraction * unlockUpgrade : Unlock a DocumentMK upgrade to a newer version * upgrade : Migrate existing Jackrabbit 2.x repository to Oak. @@ -436,18 +435,6 @@ Oak TarMK Revision Diff See the [official documentation](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html#diff). - -Oak TarMK Revision Recovery - -Lists candidates for head journal entries. Uses a read-only store, so no updates will be performed on target repository. - -$ java -jar oak-run-*.jar tarmkrecovery path/to/repository [--version-v10] - -The following options are available: - ---version-v10 - Uses V10 version repository reading (see OAK-2527) - Oak DataStore Check ---
svn commit: r1823138 - in /jackrabbit/oak/branches/1.8: ./ oak-doc/src/site/markdown/command_line.md
Author: mduerig Date: Mon Feb 5 08:35:53 2018 New Revision: 1823138 URL: http://svn.apache.org/viewvc?rev=1823138=rev Log: OAK-7241: oak-run documentation typo for checkpoints command merged r1823135 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/command_line.md Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Mon Feb 5 08:35:53 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681,1822723 +/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681,1822723,1823135 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/command_line.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/command_line.md?rev=1823138=1823137=1823138=diff == --- jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/command_line.md (original) +++ jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/command_line.md Mon Feb 5 08:35:53 2018 @@ -40,7 +40,7 @@ The `` option is the target dire To list the checkpoints of a repository, use: -java -mx4g -jar oak-run-*.jar checkpoint +java -mx4g -jar oak-run-*.jar checkpoints When using the (default) Tar storage, the `` setting is the path to the directory that contains the segment (data*.tar) files. @@ -51,7 +51,7 @@ The oak-run compact operation may be use java -mx4g -jar oak-run-*.jar compact -It makes sense to find and remove the old checkpoints, using the `checkpoint` +It makes sense to find and remove the old checkpoints, using the `checkpoints` command described above. Index Management @@ -60,7 +60,7 @@ The oak-run index operation is used to [ System properties supported by the oak-run -When performing operations on the SegmentNodeStore (eg. backup, checkpoint, compact), it's +When performing operations on the SegmentNodeStore (eg. backup, checkpoints, compact), it's possible to tweak various system properties to get the optimal performance. These are: * `cache` - cache size for the SegmentNodeStore (default: `256`), @@ -70,7 +70,7 @@ possible to tweak various system propert Example: -java -Dtar.memoryMapped=true -mx4g -jar oak-run-*.jar checkpoint +java -Dtar.memoryMapped=true -mx4g -jar oak-run-*.jar checkpoints Analyzing Thread Dumps
svn commit: r1823135 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md
Author: mduerig Date: Mon Feb 5 08:30:16 2018 New Revision: 1823135 URL: http://svn.apache.org/viewvc?rev=1823135=rev Log: OAK-7241: oak-run documentation typo for "checkpoints" command Replaced checkpoint to checkpoints. Credits to Matt Ryan for the patch. Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md?rev=1823135=1823134=1823135=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/command_line.md Mon Feb 5 08:30:16 2018 @@ -40,7 +40,7 @@ The `` option is the target dire To list the checkpoints of a repository, use: -java -mx4g -jar oak-run-*.jar checkpoint +java -mx4g -jar oak-run-*.jar checkpoints When using the (default) Tar storage, the `` setting is the path to the directory that contains the segment (data*.tar) files. @@ -51,7 +51,7 @@ The oak-run compact operation may be use java -mx4g -jar oak-run-*.jar compact -It makes sense to find and remove the old checkpoints, using the `checkpoint` +It makes sense to find and remove the old checkpoints, using the `checkpoints` command described above. Index Management @@ -60,7 +60,7 @@ The oak-run index operation is used to [ System properties supported by the oak-run -When performing operations on the SegmentNodeStore (eg. backup, checkpoint, compact), it's +When performing operations on the SegmentNodeStore (eg. backup, checkpoints, compact), it's possible to tweak various system properties to get the optimal performance. These are: * `cache` - cache size for the SegmentNodeStore (default: `256`), @@ -70,7 +70,7 @@ possible to tweak various system propert Example: -java -Dtar.memoryMapped=true -mx4g -jar oak-run-*.jar checkpoint +java -Dtar.memoryMapped=true -mx4g -jar oak-run-*.jar checkpoints Analyzing Thread Dumps
svn commit: r1822939 - /jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md
Author: mduerig Date: Fri Feb 2 10:40:17 2018 New Revision: 1822939 URL: http://svn.apache.org/viewvc?rev=1822939=rev Log: OAK-5506: add a note about how we currently treat 'broken' Java strings Typo Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md?rev=1822939=1822938=1822939=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/constraints.md Fri Feb 2 10:40:17 2018 @@ -45,7 +45,7 @@ Finally, the chosen persistence implemen Due to the way Java represents characters in strings, not every String is a valid sequence of Unicode code points. This is because *two* characters are needed to represent Unicode -"suuplementary characters". If these "surrogate" characters do not appear as a wellformed +"supplementary characters". If these "surrogate" characters do not appear as a well formed pair, the Java string can not be serialized to a sequence of Unicode characters, nor to a byte sequence (using UTF-8 character encoding).
svn commit: r1822660 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Author: mduerig
Date: Tue Jan 30 16:14:27 2018
New Revision: 1822660
URL: http://svn.apache.org/viewvc?rev=1822660=rev
Log:
OAK-7215: Add configurable repository size cap to SegmentOverflowExceptionIT
Adjust Javadoc
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java?rev=1822660=1822659=1822660=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Tue Jan 30 16:14:27 2018
@@ -57,7 +57,7 @@ import org.slf4j.LoggerFactory;
*
*
*If you only want to run this test:
- * {@code mvn verify -Dsurefire.skip.ut=true -PintegrationTesting
-Dit.test=SegmentOverflowExceptionIT}
+ * {@code mvn verify -DfailIfNoTests=false -DskipTests -PintegrationTesting
-Dtest=SegmentOverflowExceptionIT}
*
*/
public class SegmentOverflowExceptionIT {
svn commit: r1822659 - /jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Author: mduerig
Date: Tue Jan 30 16:00:45 2018
New Revision: 1822659
URL: http://svn.apache.org/viewvc?rev=1822659=rev
Log:
OAK-7215: Add configurable repository size cap to SegmentOverflowExceptionIT
Use -Dmax-repo-size=s to specify a max repo size of s megabytes
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java?rev=1822659=1822658=1822659=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentOverflowExceptionIT.java
Tue Jan 30 16:00:45 2018
@@ -19,6 +19,7 @@
package org.apache.jackrabbit.oak.segment;
+import static java.lang.System.getProperty;
import static org.apache.commons.lang.RandomStringUtils.randomAlphabetic;
import static
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.fileStoreBuilder;
import static org.junit.Assume.assumeTrue;
@@ -27,6 +28,8 @@ import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.util.Random;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
import com.google.common.collect.Iterables;
import org.apache.jackrabbit.oak.api.Blob;
@@ -36,6 +39,7 @@ import org.apache.jackrabbit.oak.spi.com
import org.apache.jackrabbit.oak.spi.gc.GCMonitor;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeStore;
+import org.apache.jackrabbit.oak.stats.DefaultStatisticsProvider;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -47,21 +51,24 @@ import org.slf4j.LoggerFactory;
* Tests verifying if the repository gets corrupted or not: {@code OAK-2662
SegmentOverflowException in HeavyWriteIT on Jenkins}
*
* This test will run for one hour unless it fails, thus it is
disabled by default. On the
- * command line specify {@code -DSegmentOverflowExceptionIT=true} to enable
it. To specify a different
- * time out {@code t} value use {@code -Dtimeout=t}
+ * command line specify {@code -Dtest=SegmentOverflowExceptionIT} to enable
it. Use {@code -Dtimeout=t}
+ * to specify a different timeout {@code t} in milliseconds. Use {@code
-Dmax-repo-size=s} to specify
+ * a maximal repository size {@code s} in megabytes above which to stop the
test.
*
*
*If you only want to run this test:
- * {@code mvn verify -Dsurefire.skip.ut=true -PintegrationTesting
-Dit.test=SegmentOverflowExceptionIT -DSegmentOverflowExceptionIT=true}
+ * {@code mvn verify -Dsurefire.skip.ut=true -PintegrationTesting
-Dit.test=SegmentOverflowExceptionIT}
*
*/
public class SegmentOverflowExceptionIT {
private static final Logger LOG = LoggerFactory
.getLogger(SegmentOverflowExceptionIT.class);
-private static final boolean ENABLED = Boolean
-.getBoolean(SegmentOverflowExceptionIT.class.getSimpleName());
+private static final boolean ENABLED =
+
SegmentOverflowExceptionIT.class.getSimpleName().equals(getProperty("test"));
private static final long TIMEOUT = Long
.getLong("timeout", 60*60*1000);
+private static final long MAX_REPO_SIZE = 1024L * 1024L * Integer
+.getInteger("max-repo-size", Integer.MAX_VALUE);
private final Random rnd = new Random();
@@ -93,12 +100,16 @@ public class SegmentOverflowExceptionIT
@Test
public void run() throws Exception {
-FileStore fileStore =
fileStoreBuilder(getFileStoreFolder()).withGCMonitor(gcMonitor).build();
-try {
+ScheduledExecutorService executor =
Executors.newSingleThreadScheduledExecutor();
+try (FileStore fileStore = fileStoreBuilder(getFileStoreFolder())
+.withGCMonitor(gcMonitor)
+.withStatisticsProvider(new
DefaultStatisticsProvider(executor))
+.build()) {
final SegmentNodeStore nodeStore =
SegmentNodeStoreBuilders.builder(fileStore).build();
long start = System.currentTimeMillis();
int snfeCount = 0;
-while (System.currentTimeMillis() - start < TIMEOUT) {
+while (System.currentTimeMillis() - start < TIMEOUT &&
+ fileStore.getStats().getApproximateSize() < MAX_REPO_SIZE) {
try {
NodeBuilder root = nodeStore.getRoot().builder();
while (rnd.nextInt(100) != 0) {
@@ -119,7 +1
svn commit: r1821841 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ta
Author: mduerig
Date: Mon Jan 22 11:33:16 2018
New Revision: 1821841
URL: http://svn.apache.org/viewvc?rev=1821841=rev
Log:
OAK-7132: SNFE after full compaction
merged r1821249
Added:
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tar/GCGenerationTest.java
- copied unchanged from r1821249,
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/tar/GCGenerationTest.java
Modified:
jackrabbit/oak/branches/1.8/ (props changed)
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/Reclaimers.java
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/GCGeneration.java
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/file/ReclaimersTest.java
Propchange: jackrabbit/oak/branches/1.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Jan 22 11:33:16 2018
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
+/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821249,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java?rev=1821841=1821840=1821841=diff
==
---
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
(original)
+++
jackrabbit/oak/branches/1.8/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java
Mon Jan 22 11:33:16 2018
@@ -30,6 +30,8 @@ import static org.apache.jackrabbit.oak.
import static
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.EMPTY_NODE;
import static
org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder.defaultSegmentWriterBuilder;
import static org.apache.jackrabbit.oak.segment.SegmentId.isDataSegmentId;
+import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.FULL;
+import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.TAIL;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCStatus.CLEANUP;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCStatus.COMPACTION;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCStatus.COMPACTION_FORCE_COMPACT;
@@ -37,6 +39,7 @@ import static org.apache.jackrabbit.oak.
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCStatus.ESTIMATION;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCStatus.IDLE;
import static
org.apache.jackrabbit.oak.segment.file.PrintableBytes.newPrintableBytes;
+import static
org.apache.jackrabbit.oak.segment.file.Reclaimers.newOldReclaimer;
import static
org.apache.jackrabbit.oak.segment.file.TarRevisions.EXPEDITE_OPTION;
import static org.apache.jackrabbit.oak.segment.file.TarRevisions.timeout;
@@ -76,6 +79,7 @@ import org.apache.jackrabbit.oak.segment
import org.apache.jackrabbit.oak.segment.SegmentWriter;
import org.apache.jackrabbit.oak.segment.WriterCacheManager;
import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions;
+import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType;
import org.apache.jackrabbit.oak.segment.file.GCJournal.GCJournalEntry;
import org.apache.jackrabbit.oak.segment.file.ShutDown.ShutDownCloser;
import org.apache.jackrabbit.oak.segment.file.tar.CleanupContext;
@@ -384,11 +388,7 @@ public class FileStore extends AbstractF
*/
public void cleanup() throws IOException {
try (ShutDownCloser ignored = shutDown.keepAlive()) {
-fileReaper.add(garbageCollector.cleanup(CompactionResult.skipped(
-getGcGeneration(),
-garbageCollector.gcOptions,
-revisions.getHead()
-)));
+fileReaper.add(garbageCollector.cleanup());
}
}
@@ -576,6 +576,15 @@ public class FileStore extends AbstractF
svn commit: r1821839 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java oak-segment-tar/src/test/java/org/apache/jackrabbit/
Author: mduerig
Date: Mon Jan 22 11:32:02 2018
New Revision: 1821839
URL: http://svn.apache.org/viewvc?rev=1821839=rev
Log:
OAK-7132: SNFE after full compaction
merged r1820729
Modified:
jackrabbit/oak/branches/1.8/ (props changed)
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionMBean.java
Propchange: jackrabbit/oak/branches/1.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Jan 22 11:32:02 2018
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
+/jackrabbit/oak/trunk:1820660-1820661,1820729,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java?rev=1821839=1821838=1821839=diff
==
---
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
(original)
+++
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentCompactionIT.java
Mon Jan 22 11:32:02 2018
@@ -34,6 +34,8 @@ import static java.util.concurrent.TimeU
import static java.util.concurrent.TimeUnit.MINUTES;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.apache.commons.lang.RandomStringUtils.randomAlphabetic;
+import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.FULL;
+import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType.TAIL;
import static
org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.defaultGCOptions;
import static
org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.fileStoreBuilder;
import static org.junit.Assert.assertNotNull;
@@ -58,6 +60,7 @@ import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@@ -85,6 +88,7 @@ import org.apache.jackrabbit.oak.plugins
import org.apache.jackrabbit.oak.plugins.commit.DefaultThreeWayConflictHandler;
import org.apache.jackrabbit.oak.plugins.metric.MetricStatisticsProvider;
import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions;
+import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions.GCType;
import org.apache.jackrabbit.oak.segment.compaction.SegmentRevisionGC;
import org.apache.jackrabbit.oak.segment.compaction.SegmentRevisionGCMBean;
import org.apache.jackrabbit.oak.segment.file.FileStore;
@@ -141,6 +145,9 @@ public class SegmentCompactionIT {
private final ListeningScheduledExecutorService scheduler =
listeningDecorator(executor);
private final FileStoreGCMonitor fileStoreGCMonitor = new
FileStoreGCMonitor(Clock.SIMPLE);
private final TestGCMonitor gcMonitor = new
TestGCMonitor(fileStoreGCMonitor);
+private final SegmentGCOptions gcOptions = defaultGCOptions()
+.setEstimationDisabled(true)
+.setForceTimeout(3600);
private final Set<Future> writers = newConcurrentHashSet();
private final Set<Future> readers = newConcurrentHashSet();
private final Set<Future> references = newConcurrentHashSet();
@@ -167,7 +174,9 @@ public class SegmentCompactionIT {
private volatile int nodeAddRatio = 40;
private volatile int addStringRatio = 20;
private volatile int addBinaryRatio = 0;
+private final AtomicInteger compactionCount = new AtomicInteger();
private volatile int compactionInterval = 2;
+private volatile int fullCompactionCycle = 4;
private volatile int maxCheckpoints = 2;
private volatile int checkpointInterval = 10;
private volatile boolean stopping;
@@ -237,9 +246,6 @@ public class SegmentCompactionIT {
ScheduledExecutorService executor =
Executors.newSingleThreadScheduledExecutor();
MetricStatisticsProvider statisticsProvider = new
MetricStatisticsProvider(mBe
svn commit: r1821840 - in /jackrabbit/oak/branches/1.8: ./ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Author: mduerig
Date: Mon Jan 22 11:32:35 2018
New Revision: 1821840
URL: http://svn.apache.org/viewvc?rev=1821840=rev
Log:
OAK-7132: SNFE after full compaction
merged r1820734
Modified:
jackrabbit/oak/branches/1.8/ (props changed)
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Propchange: jackrabbit/oak/branches/1.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Jan 22 11:32:35 2018
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1820660-1820661,1820729,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
+/jackrabbit/oak/trunk:1820660-1820661,1820729,1820734,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java?rev=1821840=1821839=1821840=diff
==
---
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
(original)
+++
jackrabbit/oak/branches/1.8/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/CompactionAndCleanupIT.java
Mon Jan 22 11:32:35 2018
@@ -84,6 +84,7 @@ import org.apache.jackrabbit.oak.stats.C
import org.apache.jackrabbit.oak.stats.DefaultStatisticsProvider;
import org.apache.jackrabbit.oak.stats.StatisticsProvider;
import org.junit.BeforeClass;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -1410,6 +1411,7 @@ public class CompactionAndCleanupIT {
}
}
+@Ignore("OAK-7132") // FIXME OAK-7132: SNFE after full compaction
@Test
public void latestFullCompactedStateShouldNotBeDeleted() throws Exception {
SegmentGCOptions gcOptions = defaultGCOptions()
@@ -1417,30 +1419,54 @@ public class CompactionAndCleanupIT {
.setRetainedGenerations(2);
try (FileStore fileStore =
fileStoreBuilder(getFileStoreFolder()).withGCOptions(gcOptions).build()) {
+SegmentNodeState previousHead;
+SegmentNodeState head = fileStore.getHead();
// Create a full, self consistent head state. This state will be
the
-// base for the following tail compactions. This increments the
-// full generation.
-
+// base for the following tail compactions. This increments the
full generation.
fileStore.fullGC();
-traverse(fileStore.getHead());
+previousHead = head;
+head = fileStore.getHead();
+
+// retainedGeneration = 2 -> the full compacted head and the
previous uncompacted head must
+// still be available.
+traverse(previousHead);
+traverse(head);
// Create a tail head state on top of the previous full state. This
-// increments the generation, but leaves the full generation
-// untouched.
+// increments the generation, but leaves the full generation
untouched.
+fileStore.tailGC();
+previousHead = head;
+head = fileStore.getHead();
+// retainedGeneration = 2 -> the tail compacted head and the
previous uncompacted head must
+// still be available.
+traverse(previousHead);
+traverse(head);
+
+// Create a tail state on top of the previous tail state. This
increments the generation,
+// but leaves the full generation untouched. This brings this
generations two generations
+// away from the latest full head state. Still, the full head
state will not be deleted
+// because doing so would generate an invalid repository at risk
of SegmentNotFoundException.
fileStore.tailGC();
-traverse(fileStore.getHead());
+previousHead = head;
+head = fileStore.getHead();
-// Create a tail state on top of the previous tail state. This
-// increments the generation, but leaves the full generation
-// untouched. This brings this generations two generations away
from
-// the latest full head state. Still, the full head state will not
-// be deleted because doing so would generate an invalid repository
-
svn commit: r1821682 - in /jackrabbit/oak/branches/1.8: ./ oak-doc/src/site/markdown/articles.md oak-doc/src/site/markdown/nodestore/segment/overview.md
Author: mduerig Date: Fri Jan 19 16:41:46 2018 New Revision: 1821682 URL: http://svn.apache.org/viewvc?rev=1821682=rev Log: OAK-7172: Document TarMK specific MBeans merged r1821681 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/articles.md jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Jan 19 16:41:46 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668 +/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668,1821681 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/articles.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/articles.md?rev=1821682=1821681=1821682=diff == --- jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/articles.md (original) +++ jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/articles.md Fri Jan 19 16:41:46 2018 @@ -20,6 +20,7 @@ This page contains links to the latest articles and presentations on Jackrabbit Oak. It is meant as an entry point for any newcommer to Jackrabbit Oak +* Valentin Olteanu, Michael Duerig, [TarMK: Facts and Figures](https://adapt.to/2017/en/schedule/tarmk--facts-and-figures.html) * Michael Duerig, [Into the tar pit: a TarMK deep dive](https://adapt.to/2016/en/schedule/into-the-tar-pit--a-tarmk-deep-dive.html) (September 2016) adaptTo() 2016 * Michael Duerig, [Avoiding and dealing with conflicting updates in Oak](https://adapt.to/2015/en/schedule/avoiding-and-dealing-with-conflicting-updates-in-oak.html) (September 2015) adaptTo() 2015 * Davide Giannella, [Scaling the query with Oak](https://adapt.to/2015/en/schedule/scaling-the-query-with-oak.html) (September 2015) adaptTo() 2015 Modified: jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1821682=1821681=1821682=diff == --- jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md (original) +++ jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md Fri Jan 19 16:41:46 2018 @@ -23,25 +23,7 @@ * [Estimation, Compaction and Cleanup](#estimation-compaction-cleanup) * [Offline Garbage Collection](#offline-garbage-collection) * [Online Garbage Collection](#online-garbage-collection) -* [Monitoring the log](#monitoring-the-log) -* [When did garbage collection start?](#when-did-garbage-collection-start) -* [When did estimation start?](#when-did-estimation-start) -* [Is estimation disabled?](#is-estimation-disabled) -* [Was estimation cancelled?](#was-estimation-cancelled) -* [When did estimation complete?](#when-did-estimation-complete) -* [When did compaction start?](#when-did-compaction-start) -* [What is the compaction type?](#what-is-the-compaction-type) -* [Is compaction disabled?](#is-compaction-disabled) -* [Was compaction cancelled?](#was-compaction-cancelled) -* [When did compaction complete?](#when-did-compaction-complete) -* [How does compaction work with concurrent writes?](#how-does-compaction-works-with-concurrent-writes) -* [How does compaction deal with checkpoints?](#how-does-compaction-deal-with-checkpoints) -* [When did clean-up start?](#when-did-cleanup-start) -* [Was cleanup cancelled?](#was-cleanup-cancelled) -* [When did cleanup complete?](#when-did-cleanup-complete) -* [What happened during cleanup?](#what-happened-during-cleanup) -* [Monitoring via JMX](#monitoring-via-jmx) -* [SegmentRevisionGarbageCollection](#SegmentRevisionGarbageCollection) +* [Monitoring](#monitoring) * [Tools](#tools) * [Backup](#backup) * [Restore](#restore) @@ -437,13 +419,170 @@ Removed files data0a.tar,data1a. The output of this message can vary. It depends on the amount of segments that were cleaned up, on how many TAR files were emptied and on how often the background activity removes unused files
svn commit: r1821681 - in /jackrabbit/oak/trunk/oak-doc/src/site/markdown: articles.md nodestore/segment/overview.md
Author: mduerig Date: Fri Jan 19 16:40:37 2018 New Revision: 1821681 URL: http://svn.apache.org/viewvc?rev=1821681=rev Log: OAK-7172: Document TarMK specific MBeans Add basic documentation of MBeans relevant to the Segment Store Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/articles.md jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/articles.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/articles.md?rev=1821681=1821680=1821681=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/articles.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/articles.md Fri Jan 19 16:40:37 2018 @@ -20,6 +20,7 @@ This page contains links to the latest articles and presentations on Jackrabbit Oak. It is meant as an entry point for any newcommer to Jackrabbit Oak +* Valentin Olteanu, Michael Duerig, [TarMK: Facts and Figures](https://adapt.to/2017/en/schedule/tarmk--facts-and-figures.html) * Michael Duerig, [Into the tar pit: a TarMK deep dive](https://adapt.to/2016/en/schedule/into-the-tar-pit--a-tarmk-deep-dive.html) (September 2016) adaptTo() 2016 * Michael Duerig, [Avoiding and dealing with conflicting updates in Oak](https://adapt.to/2015/en/schedule/avoiding-and-dealing-with-conflicting-updates-in-oak.html) (September 2015) adaptTo() 2015 * Davide Giannella, [Scaling the query with Oak](https://adapt.to/2015/en/schedule/scaling-the-query-with-oak.html) (September 2015) adaptTo() 2015 Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1821681=1821680=1821681=diff == --- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md (original) +++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Fri Jan 19 16:40:37 2018 @@ -23,25 +23,7 @@ * [Estimation, Compaction and Cleanup](#estimation-compaction-cleanup) * [Offline Garbage Collection](#offline-garbage-collection) * [Online Garbage Collection](#online-garbage-collection) -* [Monitoring the log](#monitoring-the-log) -* [When did garbage collection start?](#when-did-garbage-collection-start) -* [When did estimation start?](#when-did-estimation-start) -* [Is estimation disabled?](#is-estimation-disabled) -* [Was estimation cancelled?](#was-estimation-cancelled) -* [When did estimation complete?](#when-did-estimation-complete) -* [When did compaction start?](#when-did-compaction-start) -* [What is the compaction type?](#what-is-the-compaction-type) -* [Is compaction disabled?](#is-compaction-disabled) -* [Was compaction cancelled?](#was-compaction-cancelled) -* [When did compaction complete?](#when-did-compaction-complete) -* [How does compaction work with concurrent writes?](#how-does-compaction-works-with-concurrent-writes) -* [How does compaction deal with checkpoints?](#how-does-compaction-deal-with-checkpoints) -* [When did clean-up start?](#when-did-cleanup-start) -* [Was cleanup cancelled?](#was-cleanup-cancelled) -* [When did cleanup complete?](#when-did-cleanup-complete) -* [What happened during cleanup?](#what-happened-during-cleanup) -* [Monitoring via JMX](#monitoring-via-jmx) -* [SegmentRevisionGarbageCollection](#SegmentRevisionGarbageCollection) +* [Monitoring](#monitoring) * [Tools](#tools) * [Backup](#backup) * [Restore](#restore) @@ -437,13 +419,170 @@ Removed files data0a.tar,data1a. The output of this message can vary. It depends on the amount of segments that were cleaned up, on how many TAR files were emptied and on how often the background activity removes unused files. - Monitoring via JMX + Monitoring The Segment Store exposes certain pieces of information via JMX. This allows clients to easily access some statistics about the Segment Store, and connect the Segment Store to whatever monitoring infrastructure is in place. Moreover, JMX can be useful to execute some low-level operations in a manual fashion. -# SegmentRevisionGarbageCollection +* Each session exposes an [SessionMBean](#SessionMBean) instance, which contains counters like the number and rate of reads and writes to the session. +* The [RepositoryStatsMBean](#RepositoryStatsMBean) exposes endpoints to monitor the number of open sessions, the session login rate, the overall read and write load across all sessions, the overall read and write timings across all
svn commit: r1821671 - in /jackrabbit/oak/branches/1.8: ./ oak-doc/src/site/markdown/osgi_config.md
Author: mduerig Date: Fri Jan 19 15:41:08 2018 New Revision: 1821671 URL: http://svn.apache.org/viewvc?rev=1821671=rev Log: OAK-7075: Document oak-run compact arguments and system properties merged r1821668 Modified: jackrabbit/oak/branches/1.8/ (props changed) jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/osgi_config.md Propchange: jackrabbit/oak/branches/1.8/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Jan 19 15:41:08 2018 @@ -1,3 +1,3 @@ /jackrabbit/oak/branches/1.0:1665962 -/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665 +/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665,1821668 /jackrabbit/trunk:1345480 Modified: jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/osgi_config.md URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/osgi_config.md?rev=1821671=1821670=1821671=diff == --- jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/osgi_config.md (original) +++ jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/osgi_config.md Fri Jan 19 15:41:08 2018 @@ -47,7 +47,7 @@ The first of these configurations, ident This implementation is the newest and the only actively maintained one, and should be used unless some legacy use case needs to be supported. The second and last configuration, identified by `org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService`, refers to the old implementation of the Node Store provided by the `oak-segment` bundle. -This implementation has been deprecated, will not receive any further improvements and should not be used, if possible. +This implementation has been deprecated and removed in Oak 1.8. It will not receive any further improvements and should not be used in new projects. # org.apache.jackrabbit.oak.segment.SegmentNodeStoreService @@ -114,7 +114,7 @@ compaction.disableEstimation (boolean) - If this property is set to `true`, the estimation phase of the compaction process will never run, and compaction will always be triggered for any amount of garbage in the Node Store. compaction.retainedGenerations (int) - 2 -: The number of generations to retain the Node Store. +: The number of generations to retain the Node Store. As of Oak 1.8 this option is deprecated. Its value defaults to `2` and cannot be changed by the user. compaction.memoryThreshold (int) - 15 : The percentage of heap memory that should always be free while compaction runs. @@ -139,7 +139,7 @@ Placeholders of any kind in the path are # org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService -**This configuration is deprecated. +**This configuration is deprecated and has been removed in Oak 1.8. It belongs to the legacy Node Store implementation provided by the oak-segment bundle. This implementation should not be used anymore. Instead, rely on the Node Store implementation provided by the oak-segment-tar bundle, whose configuration is described above.**
svn commit: r1821670 - in /jackrabbit/oak/branches/1.8: ./ oak-doc/src/site/markdown/nodestore/segment/overview.md
Author: mduerig
Date: Fri Jan 19 15:40:31 2018
New Revision: 1821670
URL: http://svn.apache.org/viewvc?rev=1821670=rev
Log:
OAK-7075: Document oak-run compact arguments and system properties
merged r1821665
Modified:
jackrabbit/oak/branches/1.8/ (props changed)
jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md
Propchange: jackrabbit/oak/branches/1.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Fri Jan 19 15:40:31 2018
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516
+/jackrabbit/oak/trunk:1820660-1820661,1820859,1820861,1820878,1820888,1820947,1821130,1821140-1821141,1821240,1821258,1821325,1821358,1821361-1821362,1821370,1821375,1821393,1821477,1821487,1821516,1821665
/jackrabbit/trunk:1345480
Modified:
jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1821670=1821669=1821670=diff
==
---
jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md
(original)
+++
jackrabbit/oak/branches/1.8/oak-doc/src/site/markdown/nodestore/segment/overview.md
Fri Jan 19 15:40:31 2018
@@ -466,13 +466,17 @@ This parameter is used only if compactio
* **RetainedGenerations (int)**
How many generations should be preserved when cleaning up the Segment Store.
When the cleanup phase runs, only the latest `RetainedGenerations` generations
are kept intact.
-Older generations will be deleted.
+Older generations will be deleted. *Deprecated*: as of Oak 1.8 this value is
fixed to 2 generations and cannot be modified.
* **GcSizeDeltaEstimation (long)**
The size (in bytes) of new content added to the repository since the end of
the last garbage collection that would trigger another garbage collection run.
This parameter influences the behaviour of the estimation phase.
* **EstimationDisabled (boolean)**
Determines if the estimation phase is disabled.
If this parameter is set to `true`, the estimation phase will be skipped and
compaction will run unconditionally.
+* **GCType ("FULL" or "TAIL")**
+Determines the type of the garbage collection that should run when invoking
the `startRevisionGC` operation.
+* **RevisionGCProgressLog (long)**
+The number of processed nodes after which a progress message is logged. `-1`
indicates no logging.
* **MemoryThreshold (int)**
A number between `0` and `100` that represents the percentage of heap memory
that should always be free during compaction.
If the amount of free memory falls below the provided percentage, compaction
will be interrupted.
@@ -494,6 +498,14 @@ The last log message produced during gar
* **Status (string)**
The current status of the garbage collection process.
This property can assume the values `idle`, `estimation`, `compaction`,
`compaction-retry-N` (where `N` is the number of the current retry iteration),
`compaction-force-compact` and `cleanup`.
+* **RevisionGCRunning (boolean)**
+Indicates whether online revision garbage collection is currently running.
+* **CompactedNodes (long)**
+The number of compacted nodes during the previous garbage collection
+* **EstimatedCompactableNodes (long)**
+The estimated number of nodes to compact during the next garbage collection.
`-1` indicates an estimated value is not available.
+* **EstimatedRevisionGCCompletion (int)**
+Estimated percentage completed for the current garbage collection run. `-1`
indicates an estimated percentage is not available.
The `SegmentRevisionGarbageCollection` MBean also exposes the following
management operations.
