[jira] [Resolved] (OAK-6924) Build Jackrabbit Oak #961 failed
[ https://issues.apache.org/jira/browse/OAK-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh resolved OAK-6924. Resolution: Fixed Reverted fix of OAK-6838 which broke this. > Build Jackrabbit Oak #961 failed > > > Key: OAK-6924 > URL: https://issues.apache.org/jira/browse/OAK-6924 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson > > No description is provided > The build Jackrabbit Oak #961 has failed. > First failed run: [Jackrabbit Oak > #961|https://builds.apache.org/job/Jackrabbit%20Oak/961/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/961/console] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (OAK-6838) IS NOT NULL condition for relative properties not working as expected
[
https://issues.apache.org/jira/browse/OAK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247003#comment-16247003
]
Vikas Saurabh edited comment on OAK-6838 at 11/10/17 7:00 AM:
--
-Fixed in trunk at [r1814817|https://svn.apache.org/r1814817].- Reverted the
fix at [r1814821|https://svn.apache.org/r1814821] as it broke
{{org.apache.jackrabbit.oak.jcr.query.QueryTest.relativeNotExistsProperty[SegmentTar]}}
\[0]
\[0]: https://builds.apache.org/job/Jackrabbit%20Oak/961/
was (Author: catholicon):
Fixed in trunk at [r1814817|https://svn.apache.org/r1814817].
> IS NOT NULL condition for relative properties not working as expected
> -
>
> Key: OAK-6838
> URL: https://issues.apache.org/jira/browse/OAK-6838
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6838.patch
>
>
> It seems that a query like
> {noformat}
> SELECT * FROM [nt:base] WHERE [a/b] IS NULL
> {noformat}
> should return paths where there exists a child {{a}} without a property {{b}}
> (this works as expected)
> But, it should also return paths where there is not child {{a}} in the first
> place.
> [~tmueller], does this sound a reasonable expectation?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (OAK-6838) IS NOT NULL condition for relative properties not working as expected
[
https://issues.apache.org/jira/browse/OAK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh reopened OAK-6838:
> IS NOT NULL condition for relative properties not working as expected
> -
>
> Key: OAK-6838
> URL: https://issues.apache.org/jira/browse/OAK-6838
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6838.patch
>
>
> It seems that a query like
> {noformat}
> SELECT * FROM [nt:base] WHERE [a/b] IS NULL
> {noformat}
> should return paths where there exists a child {{a}} without a property {{b}}
> (this works as expected)
> But, it should also return paths where there is not child {{a}} in the first
> place.
> [~tmueller], does this sound a reasonable expectation?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6838) IS NOT NULL condition for relative properties not working as expected
[
https://issues.apache.org/jira/browse/OAK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-6838:
---
Fix Version/s: (was: 1.7.12)
> IS NOT NULL condition for relative properties not working as expected
> -
>
> Key: OAK-6838
> URL: https://issues.apache.org/jira/browse/OAK-6838
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6838.patch
>
>
> It seems that a query like
> {noformat}
> SELECT * FROM [nt:base] WHERE [a/b] IS NULL
> {noformat}
> should return paths where there exists a child {{a}} without a property {{b}}
> (this works as expected)
> But, it should also return paths where there is not child {{a}} in the first
> place.
> [~tmueller], does this sound a reasonable expectation?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6924) Build Jackrabbit Oak #961 failed
Hudson created OAK-6924: --- Summary: Build Jackrabbit Oak #961 failed Key: OAK-6924 URL: https://issues.apache.org/jira/browse/OAK-6924 Project: Jackrabbit Oak Issue Type: Bug Components: continuous integration Reporter: Hudson No description is provided The build Jackrabbit Oak #961 has failed. First failed run: [Jackrabbit Oak #961|https://builds.apache.org/job/Jackrabbit%20Oak/961/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/961/console] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6838) IS NOT NULL condition for relative properties not working as expected
[
https://issues.apache.org/jira/browse/OAK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-6838.
Resolution: Fixed
Fix Version/s: 1.7.12
Fixed in trunk at [r1814817|https://svn.apache.org/r1814817].
> IS NOT NULL condition for relative properties not working as expected
> -
>
> Key: OAK-6838
> URL: https://issues.apache.org/jira/browse/OAK-6838
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.8, 1.7.12
>
> Attachments: OAK-6838.patch
>
>
> It seems that a query like
> {noformat}
> SELECT * FROM [nt:base] WHERE [a/b] IS NULL
> {noformat}
> should return paths where there exists a child {{a}} without a property {{b}}
> (this works as expected)
> But, it should also return paths where there is not child {{a}} in the first
> place.
> [~tmueller], does this sound a reasonable expectation?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-3087) [oak-mongo.js] Add utility to cleanup hidden structure under disabled indices
[ https://issues.apache.org/jira/browse/OAK-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246963#comment-16246963 ] Vikas Saurabh commented on OAK-3087: Unscheduling from 1.8. > [oak-mongo.js] Add utility to cleanup hidden structure under disabled indices > - > > Key: OAK-3087 > URL: https://issues.apache.org/jira/browse/OAK-3087 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Vikas Saurabh >Assignee: Vikas Saurabh >Priority: Minor > Attachments: > 0001-update-removeDescendantsAndSelf-for-upwards-removal.patch, > 0002-OAK-3087-Add-some-methods-to-find-disabled-indices-a.patch, > cleanupDisbaledIndices-take2.patch, cleanupDisbaledIndices-take3.patch, > removeDescendantsAndSelf-api-take2.patch, > removeDescendantsAndSelf-api-take3.patch > > > While disabling property indices, avoids usage of those indices. But, they > still maintain the data already stored under them. That data would keep on > consuming storage space without serving any purpose. Also, it'd pile on > mongo's id index. > While one can delete index definition node to clear these nodes up -- but, > it'd be really slow and a JCR based deleted would first create a HUGE commit > while marking all documents under it as deleted. And, then the actual > deletion would happen in next revision GC after 24 hours have past. > Hence, it might be beneficial to have a low level api in oak-mongo.js, which > simply removes the document from mongo altogether. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-3087) [oak-mongo.js] Add utility to cleanup hidden structure under disabled indices
[ https://issues.apache.org/jira/browse/OAK-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh updated OAK-3087: --- Fix Version/s: (was: 1.8) > [oak-mongo.js] Add utility to cleanup hidden structure under disabled indices > - > > Key: OAK-3087 > URL: https://issues.apache.org/jira/browse/OAK-3087 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Vikas Saurabh >Assignee: Vikas Saurabh >Priority: Minor > Attachments: > 0001-update-removeDescendantsAndSelf-for-upwards-removal.patch, > 0002-OAK-3087-Add-some-methods-to-find-disabled-indices-a.patch, > cleanupDisbaledIndices-take2.patch, cleanupDisbaledIndices-take3.patch, > removeDescendantsAndSelf-api-take2.patch, > removeDescendantsAndSelf-api-take3.patch > > > While disabling property indices, avoids usage of those indices. But, they > still maintain the data already stored under them. That data would keep on > consuming storage space without serving any purpose. Also, it'd pile on > mongo's id index. > While one can delete index definition node to clear these nodes up -- but, > it'd be really slow and a JCR based deleted would first create a HUGE commit > while marking all documents under it as deleted. And, then the actual > deletion would happen in next revision GC after 24 hours have past. > Hence, it might be beneficial to have a low level api in oak-mongo.js, which > simply removes the document from mongo altogether. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-2847) Dependency cleanup
[ https://issues.apache.org/jira/browse/OAK-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246962#comment-16246962 ] Vikas Saurabh commented on OAK-2847: [~mduerig], I guess we can retire this issue!? > Dependency cleanup > --- > > Key: OAK-2847 > URL: https://issues.apache.org/jira/browse/OAK-2847 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Michael Dürig >Assignee: Vikas Saurabh > Labels: technical_debt > Fix For: 1.8 > > > Early in the next release cycle we should go through the list of Oak's > dependencies and decide whether we have candidates we want to upgrade and > remove orphaned dependencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-3883) Avoid commit from too far in the future (due to clock skews) to go through
[ https://issues.apache.org/jira/browse/OAK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh updated OAK-3883: --- Fix Version/s: (was: 1.9) > Avoid commit from too far in the future (due to clock skews) to go through > -- > > Key: OAK-3883 > URL: https://issues.apache.org/jira/browse/OAK-3883 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Vikas Saurabh >Assignee: Vikas Saurabh > Labels: resilience > > Following up [discussion|http://markmail.org/message/m5jk5nbby77nlqs5] \[0] > to avoid bad commits due to misbehaving clocks. Points from the discussion: > * We can start self-destruct mode while updating lease > * Revision creation should check that newly created revision isn't beyond > leaseEnd time > * Implementation done for OAK-2682 might be useful > [0]: http://markmail.org/message/m5jk5nbby77nlqs5 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-3883) Avoid commit from too far in the future (due to clock skews) to go through
[ https://issues.apache.org/jira/browse/OAK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh updated OAK-3883: --- Fix Version/s: (was: 1.8) 1.9 > Avoid commit from too far in the future (due to clock skews) to go through > -- > > Key: OAK-3883 > URL: https://issues.apache.org/jira/browse/OAK-3883 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Vikas Saurabh >Assignee: Vikas Saurabh > Labels: resilience > Fix For: 1.9 > > > Following up [discussion|http://markmail.org/message/m5jk5nbby77nlqs5] \[0] > to avoid bad commits due to misbehaving clocks. Points from the discussion: > * We can start self-destruct mode while updating lease > * Revision creation should check that newly created revision isn't beyond > leaseEnd time > * Implementation done for OAK-2682 might be useful > [0]: http://markmail.org/message/m5jk5nbby77nlqs5 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-3895) Expose jmx to invalidate cache entries
[
https://issues.apache.org/jira/browse/OAK-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-3895:
---
Fix Version/s: (was: 1.8)
> Expose jmx to invalidate cache entries
> --
>
> Key: OAK-3895
> URL: https://issues.apache.org/jira/browse/OAK-3895
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
>
> We have had some issues (e.g. OAK-3634) which put incorrect entry in the some
> cache. Currently, the only way to clean cache is to restart the instance (and
> remove persistent cache files if configured).
> So, it might be useful to expose some jmx to invalidate cache without
> restarting the instance. As a first step, I think we can clean all caches
> (which is equivalent to current setup but without restart). We can later
> expose invalidation methods for specific cases (say, {{cacheType, path, rev}}
> as applicable).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-5143) Augmented query terms due to FulltextQueryTermsProvider tend to get more weight while scoring results
[
https://issues.apache.org/jira/browse/OAK-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-5143:
---
Fix Version/s: (was: 1.8)
> Augmented query terms due to FulltextQueryTermsProvider tend to get more
> weight while scoring results
> -
>
> Key: OAK-5143
> URL: https://issues.apache.org/jira/browse/OAK-5143
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.4
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
>
> Current implementation of using FulltextQueryTermsProvider service(s)
> collects query terms from these objects and attaches it to full text query as
> a big or clause. So, if original query was {{a='b' OR c='d'}} and we had a
> query term provider which would add {{e='f'}} then the final query would look
> like {{(a='b' OR c='d') OR e='f'}}. This query is correct semantically but
> while scoring from lucene's perspective the clause {{e='f'}} can singly allow
> the condition to be true and hence it gets more weight (actually, the truth
> is that the first part get less weight if only one condition from first part
> is met - ie if indeed a='b' but c!='d' then the score for first part of the
> query get a multiplier of 0.5 aka coord-factor \[0]).
> \[0]:
> https://lucene.apache.org/core/4_7_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6905) Query engine: support coalesce function as in-built method
[
https://issues.apache.org/jira/browse/OAK-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-6905:
---
Summary: Query engine: support coalesce function as in-built method (was:
Query enging: support coalesce function as in-built method)
> Query engine: support coalesce function as in-built method
> --
>
> Key: OAK-6905
> URL: https://issues.apache.org/jira/browse/OAK-6905
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, lucene, query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
> Fix For: 1.8
>
>
> Sometimes the content structure stores semantically similar properties at
> different locations (due to content evolution or different locations
> genuinely represent different values).
> e.g. say a content represent a real world image as
> {{/file1.jpg/jcr:content/file}}. Where
> {{jcr:content/metadata/exif:title}} could be title of image from exif data,
> {{jcr:content/jcr:title}} could be title explicitly set by user.
> While displaying a title, it probably makes sense to show {{jcr:title}} first
> and fallback to {{exif:title}} and then probaly to node name if nothing else
> works out.
> it'd would be useful to have such overlaying be conveyed to query engine too
> for lookup, ordering (and, of course, indexing). As a method name, sql
> colesce is exactly what should serve this purpose.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6920) Build Jackrabbit Oak #956 failed
[ https://issues.apache.org/jira/browse/OAK-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246251#comment-16246251 ] Hudson commented on OAK-6920: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #960|https://builds.apache.org/job/Jackrabbit%20Oak/960/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/960/console] > Build Jackrabbit Oak #956 failed > > > Key: OAK-6920 > URL: https://issues.apache.org/jira/browse/OAK-6920 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson > > No description is provided > The build Jackrabbit Oak #956 has failed. > First failed run: [Jackrabbit Oak > #956|https://builds.apache.org/job/Jackrabbit%20Oak/956/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/956/console] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6920) Build Jackrabbit Oak #956 failed
[ https://issues.apache.org/jira/browse/OAK-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246133#comment-16246133 ] Hudson commented on OAK-6920: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #959|https://builds.apache.org/job/Jackrabbit%20Oak/959/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/959/console] > Build Jackrabbit Oak #956 failed > > > Key: OAK-6920 > URL: https://issues.apache.org/jira/browse/OAK-6920 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson > > No description is provided > The build Jackrabbit Oak #956 has failed. > First failed run: [Jackrabbit Oak > #956|https://builds.apache.org/job/Jackrabbit%20Oak/956/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/956/console] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-5519) Skip problematic binaries instead of blocking indexing
[ https://issues.apache.org/jira/browse/OAK-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller resolved OAK-5519. - Resolution: Fixed I consider this resolved, but let me know if you see problems with the current approach. > Skip problematic binaries instead of blocking indexing > -- > > Key: OAK-5519 > URL: https://issues.apache.org/jira/browse/OAK-5519 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: indexing >Reporter: Alexander Klimetschek >Assignee: Thomas Mueller > Labels: resilience > Fix For: 1.8, 1.7.12 > > > If a text extraction is blocked (weird PDF) or a blob cannot be found in the > datastore or any other error upon indexing one item from the repository that > is outside the scope of the indexer, it currently halts the indexing (lane). > Thus one item (that maybe isn't important to the users at all) can block the > indexing of other, new content (that might be important to users), and it > always requires manual intervention (which is also not easy and requires oak > experts). > Instead, the item could be remembered in a known issue list, proper warnings > given, and indexing continue. Maintenance operations should be available to > come back to reindex these, or the indexer could automatically retry after > some time. This would allow normal user activity to go on without manual > intervention, and solving the problem (if it's isolated to some binaries) can > be deferred. > I think the line should probably be drawn for binary properties. Not sure if > other JCR property types could trigger a similar issue, and if a failure in > them might actually warrant a halt, as it could lead to an "incorrect" index, > if these properties are important. But maybe the line is simply a try & catch > around "full text extraction". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (OAK-5519) Skip problematic binaries instead of blocking indexing
[
https://issues.apache.org/jira/browse/OAK-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246063#comment-16246063
]
Thomas Mueller edited comment on OAK-5519 at 11/9/17 5:07 PM:
--
http://svn.apache.org/r1814745
[~chetanm] I have incorporated your requests. Features:
* No OSGi / JMX configuration right now, but "emergency" configuration via
system properties (for example, ability to disable this feature, set
timeout,...)
* Timeout is 60 seconds.
* Timed out extraction is now stored to a file in the repository / index
directory, in a properties file named "textExtractionTimeout.properties".
Example content below. This file is read on startup (and kept fully in memory -
so better not use that mechanism with large files).
{noformat}
#Text extraction timed out for the following binaries, and will not be retried
#Thu Nov 09 12:33:52 CET 2017
405dfb76526462a6268f1aacb09359179216df423c474b3a1f578b9c567faa35\#190148=TextExtractionError
d19a28de09b655dbe099ee9e72e5bc782088994cca054062213d80b22f2ac67f\#175=TextExtractionError
251c6082691578dc1aff306a59984e1b80a79befd8465e158335c5cbfe8bb596\#399142=TextExtractionError
{noformat}
* Failed extraction is cached.
* Number of extractions that timed out can be read via JMX
(TextExtractionStatsMBean.getTimeoutCount). Each of those threads can consume
100% CPU (unless they stop at some point).
* It is using its own executor service with daemon threads. This is shut down
when stopping the service, and restarted when needed. Just one thread usually,
up to 10 (configurable), so worst case up to 900% CPU usage if 9 extractions
time out.
* Thread name is "oak binary text extractor" plus the name of the extracted
blob (similar to what it was before).
* Only binaries larger than 16 KB are extracted in a separate thread.
* A warning is logged if extraction times out.
* No change for OutOfMemory and so on (Throwable was already caught before this
patch). So this patch only affects timeouts.
was (Author: tmueller):
http://svn.apache.org/r1814745
[~chetanm] I have incorporated your requests. Features:
* No OSGi / JMX configuration right now, but "emergency" configuration via
system properties (for example, ability to disable this feature, set
timeout,...)
* Timeout is 60 seconds.
* Timed out extraction is now stored to a file in the repository / index
directory, in a properties file named "textExtractionTimeout.properties".
Example content below. This file is read on startup.
{noformat}
#Text extraction timed out for the following binaries, and will not be retried
#Thu Nov 09 12:33:52 CET 2017
405dfb76526462a6268f1aacb09359179216df423c474b3a1f578b9c567faa35\#190148=TextExtractionError
d19a28de09b655dbe099ee9e72e5bc782088994cca054062213d80b22f2ac67f\#175=TextExtractionError
251c6082691578dc1aff306a59984e1b80a79befd8465e158335c5cbfe8bb596\#399142=TextExtractionError
{noformat}
* Failed extraction is cached.
* Number of extractions that timed out can be read via JMX
(TextExtractionStatsMBean.getTimeoutCount). Each of those threads can consume
100% CPU (unless they stop at some point).
* It is using its own executor service with daemon threads. This is shut down
when stopping the service, and restarted when needed. Just one thread usually,
up to 10 (configurable), so worst case up to 900% CPU usage if 9 extractions
time out.
* Thread name is "oak binary text extractor" plus the name of the extracted
blob (similar to what it was before).
* Only binaries larger than 16 KB are extracted in a separate thread.
* A warning is logged if extraction times out.
* No change for OutOfMemory and so on (Throwable was already caught before this
patch). So this patch only affects timeouts.
> Skip problematic binaries instead of blocking indexing
> --
>
> Key: OAK-5519
> URL: https://issues.apache.org/jira/browse/OAK-5519
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: indexing
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Labels: resilience
> Fix For: 1.8, 1.7.12
>
>
> If a text extraction is blocked (weird PDF) or a blob cannot be found in the
> datastore or any other error upon indexing one item from the repository that
> is outside the scope of the indexer, it currently halts the indexing (lane).
> Thus one item (that maybe isn't important to the users at all) can block the
> indexing of other, new content (that might be important to users), and it
> always requires manual intervention (which is also not easy and requires oak
> experts).
> Instead, the item could be remembered in a known issue list, proper warnings
> given, and indexing continue. Maintenance operations should be available to
> come back to reindex these, or the indexer could automatically retry
[jira] [Updated] (OAK-5519) Skip problematic binaries instead of blocking indexing
[ https://issues.apache.org/jira/browse/OAK-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-5519: Fix Version/s: 1.7.12 > Skip problematic binaries instead of blocking indexing > -- > > Key: OAK-5519 > URL: https://issues.apache.org/jira/browse/OAK-5519 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: indexing >Reporter: Alexander Klimetschek >Assignee: Thomas Mueller > Labels: resilience > Fix For: 1.8, 1.7.12 > > > If a text extraction is blocked (weird PDF) or a blob cannot be found in the > datastore or any other error upon indexing one item from the repository that > is outside the scope of the indexer, it currently halts the indexing (lane). > Thus one item (that maybe isn't important to the users at all) can block the > indexing of other, new content (that might be important to users), and it > always requires manual intervention (which is also not easy and requires oak > experts). > Instead, the item could be remembered in a known issue list, proper warnings > given, and indexing continue. Maintenance operations should be available to > come back to reindex these, or the indexer could automatically retry after > some time. This would allow normal user activity to go on without manual > intervention, and solving the problem (if it's isolated to some binaries) can > be deferred. > I think the line should probably be drawn for binary properties. Not sure if > other JCR property types could trigger a similar issue, and if a failure in > them might actually warrant a halt, as it could lead to an "incorrect" index, > if these properties are important. But maybe the line is simply a try & catch > around "full text extraction". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-5519) Skip problematic binaries instead of blocking indexing
[
https://issues.apache.org/jira/browse/OAK-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246063#comment-16246063
]
Thomas Mueller commented on OAK-5519:
-
http://svn.apache.org/r1814745
[~chetanm] I have incorporated your requests. Features:
* No OSGi / JMX configuration right now, but "emergency" configuration via
system properties (for example, ability to disable this feature, set
timeout,...)
* Timeout is 60 seconds.
* Timed out extraction is now stored to a file in the repository / index
directory, in a properties file named "textExtractionTimeout.properties".
Example content below. This file is read on startup.
{noformat}
#Text extraction timed out for the following binaries, and will not be retried
#Thu Nov 09 12:33:52 CET 2017
405dfb76526462a6268f1aacb09359179216df423c474b3a1f578b9c567faa35\#190148=TextExtractionError
d19a28de09b655dbe099ee9e72e5bc782088994cca054062213d80b22f2ac67f\#175=TextExtractionError
251c6082691578dc1aff306a59984e1b80a79befd8465e158335c5cbfe8bb596\#399142=TextExtractionError
{noformat}
* Failed extraction is cached.
* Number of extractions that timed out can be read via JMX
(TextExtractionStatsMBean.getTimeoutCount). Each of those threads can consume
100% CPU (unless they stop at some point).
* It is using its own executor service with daemon threads. This is shut down
when stopping the service, and restarted when needed. Just one thread usually,
up to 10 (configurable), so worst case up to 900% CPU usage if 9 extractions
time out.
* Thread name is "oak binary text extractor" plus the name of the extracted
blob (similar to what it was before).
* Only binaries larger than 16 KB are extracted in a separate thread.
* A warning is logged if extraction times out.
* No change for OutOfMemory and so on (Throwable was already caught before this
patch). So this patch only affects timeouts.
> Skip problematic binaries instead of blocking indexing
> --
>
> Key: OAK-5519
> URL: https://issues.apache.org/jira/browse/OAK-5519
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: indexing
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Labels: resilience
> Fix For: 1.8
>
>
> If a text extraction is blocked (weird PDF) or a blob cannot be found in the
> datastore or any other error upon indexing one item from the repository that
> is outside the scope of the indexer, it currently halts the indexing (lane).
> Thus one item (that maybe isn't important to the users at all) can block the
> indexing of other, new content (that might be important to users), and it
> always requires manual intervention (which is also not easy and requires oak
> experts).
> Instead, the item could be remembered in a known issue list, proper warnings
> given, and indexing continue. Maintenance operations should be available to
> come back to reindex these, or the indexer could automatically retry after
> some time. This would allow normal user activity to go on without manual
> intervention, and solving the problem (if it's isolated to some binaries) can
> be deferred.
> I think the line should probably be drawn for binary properties. Not sure if
> other JCR property types could trigger a similar issue, and if a failure in
> them might actually warrant a halt, as it could lead to an "incorrect" index,
> if these properties are important. But maybe the line is simply a try & catch
> around "full text extraction".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6839) RDBDocumentStore: queries done by GC processes should not thrash the cache
[ https://issues.apache.org/jira/browse/OAK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-6839: Fix Version/s: 1.6.7 > RDBDocumentStore: queries done by GC processes should not thrash the cache > -- > > Key: OAK-6839 > URL: https://issues.apache.org/jira/browse/OAK-6839 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.8, 1.7.10, 1.6.7 > > Attachments: OAK-6839-2.diff, OAK-6839.diff > > > GC/bg processes that use the DocumentStore query API may thrash the cache > (OAK-6806). > This is relevant for Oak versions that do not yet use special APIs for GC, > such as trunk and 1.6. > We should avoid caching result documents when the query likely was initiated > by a GC process. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6839) RDBDocumentStore: queries done by GC processes should not thrash the cache
[ https://issues.apache.org/jira/browse/OAK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245899#comment-16245899 ] Julian Reschke commented on OAK-6839: - trunk: [r1812739|http://svn.apache.org/r1812739] 1.6: [r1814740|http://svn.apache.org/r1814740] > RDBDocumentStore: queries done by GC processes should not thrash the cache > -- > > Key: OAK-6839 > URL: https://issues.apache.org/jira/browse/OAK-6839 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.8, 1.7.10, 1.6.7 > > Attachments: OAK-6839-2.diff, OAK-6839.diff > > > GC/bg processes that use the DocumentStore query API may thrash the cache > (OAK-6806). > This is relevant for Oak versions that do not yet use special APIs for GC, > such as trunk and 1.6. > We should avoid caching result documents when the query likely was initiated > by a GC process. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6839) RDBDocumentStore: queries done by GC processes should not thrash the cache
[ https://issues.apache.org/jira/browse/OAK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-6839: Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 (was: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 candidate_oak_1_6) > RDBDocumentStore: queries done by GC processes should not thrash the cache > -- > > Key: OAK-6839 > URL: https://issues.apache.org/jira/browse/OAK-6839 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.8, 1.7.10, 1.6.7 > > Attachments: OAK-6839-2.diff, OAK-6839.diff > > > GC/bg processes that use the DocumentStore query API may thrash the cache > (OAK-6806). > This is relevant for Oak versions that do not yet use special APIs for GC, > such as trunk and 1.6. > We should avoid caching result documents when the query likely was initiated > by a GC process. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (OAK-6839) RDBDocumentStore: queries done by GC processes should not thrash the cache
[ https://issues.apache.org/jira/browse/OAK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-6839: Comment: was deleted (was: trunk: [r1812739|http://svn.apache.org/r1812739] ) > RDBDocumentStore: queries done by GC processes should not thrash the cache > -- > > Key: OAK-6839 > URL: https://issues.apache.org/jira/browse/OAK-6839 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.8, 1.7.10, 1.6.7 > > Attachments: OAK-6839-2.diff, OAK-6839.diff > > > GC/bg processes that use the DocumentStore query API may thrash the cache > (OAK-6806). > This is relevant for Oak versions that do not yet use special APIs for GC, > such as trunk and 1.6. > We should avoid caching result documents when the query likely was initiated > by a GC process. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6923) Update Oak trunk to Jackrabbit 2.15.8
Julian Reschke created OAK-6923: --- Summary: Update Oak trunk to Jackrabbit 2.15.8 Key: OAK-6923 URL: https://issues.apache.org/jira/browse/OAK-6923 Project: Jackrabbit Oak Issue Type: Bug Components: parent Reporter: Julian Reschke Assignee: Julian Reschke Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6917) Configuration presets for DocumentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245637#comment-16245637
]
Julian Sedding commented on OAK-6917:
-
bq. Nope, that doesn't work
A shame, but not entirely unexpected. Did you try changing the field's type to
{{Map}}? That should work (at least with DS 1.3).
>From the Declarative Services 1.3 Spec, References to Services section 112.3.3
>Field Strategy (OSGi R6 Compendium):
{quote}
For a reference with unary cardinality, a field must be of one of the following
types:
...
Map - An unmodifiable Map containing the service properties of the bound
service. This Map must additionally implement Comparable with the compareTo
method comparing service prop- erty maps using the same ordering as
ServiceReference.compareTo based upon service ranking and service id.
{quote}
> Configuration presets for DocumentNodeStoreService
> --
>
> Key: OAK-6917
> URL: https://issues.apache.org/jira/browse/OAK-6917
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch
>
>
> When Oak is deployed in an OSGi container, applications usually want to ship
> a default configuration which is different from the defaults present in Oak.
> E.g. an application may want to use a default cache size of 1G for the
> DocumentNodeStoreService instead of the default 256M. Now if a user of the
> application provides a custom configuration and does not specify the cache
> size, the value for this configuration will flip back to the Oak default of
> 256M.
> There should be a way to configure presets for the application that are
> different from the Oak defaults and then allow a user to customize the
> configuration while still respecting the presets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or CIFS which can be simply mounted locally.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
$ hdfs dfs -ls /oak/data0a.tar
Found 517 items
-rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
/oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
-rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
/oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
(...)
-rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.brf
-rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.gph
-rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.idx
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of HDFS each write involves a network latency. That's why
the SegmentWriteQueue was introduced. The segments are added to the blocking
deque, which is served by a number of the consumer threads, writing the
segments to HDFS. There's also a map UUID->Segment, which allows to return the
segments in case they are requested by the readSegment() method before they are
actually persisted. Segments are removed from the map only after a successful
write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h5. Queue recovery mode
If the HDFS write() operation fails, the segment will be re-added and the queue
is switched to an "recovery mode". In this mode, all the threads are suspended
and new segments are not accepted (active waiting). There's a single thread
which retries adding the segment with some delay. If the segment is
successfully written, the queue will back to the normal operation.
This way the unavailable HDFS service is not flooded by the requests and we're
not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h5. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments are not
persisted.
* TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
S3 are persisted, but the S1 is not.
On the other hand:
* If the TarWriter#flush() returns successfully, it means that all the accepted
segments has been persisted.
This may lead to data inconsistency - especially the second case, where we lose
a middle segment. The impact is still to be discussed.
h3. TODO
* move the implementation to its own bundle (requires OSGi support for
SegmentArchvieManager).
was:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: (was: OAK-6922.patch)
> HDFS support for the segment-tar
>
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
> Fix For: 1.9.0
>
> Attachments: OAK-6922.patch
>
>
> A HDFS implementation of the segment storage, based on the OAK-6921 work.
> h3. HDFS
> HDFS is a distributed, network file system. The most popular implementation
> is Apache Hadoop, but the HDFS is also available in the Amazon AWS and
> Microsoft Azure clouds. Despite being a file system, it requires a custom API
> to be used - it's not similar to NFS or CIFS which can be simply mounted
> locally.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> $ hdfs dfs -ls /oak/data0a.tar
> Found 517 items
> -rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
> /oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
> -rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
> /oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
> -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
> /oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
> -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
> /oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
> (...)
> -rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.brf
> -rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.gph
> -rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.idx
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of HDFS each write involves a network latency. That's
> why the SegmentWriteQueue was introduced. The segments are added to the
> blocking deque, which is served by a number of the consumer threads, writing
> the segments to HDFS. There's also a map UUID->Segment, which allows to
> return the segments in case they are requested by the readSegment() method
> before they are actually persisted. Segments are removed from the map only
> after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. TODO: queue recovery mode
> We need to handle the write failures in a better way: if the HDFS write()
> operation fails, we should re-add the segment to the queue and switch to an
> "recovery mode". In this mode, all the threads are suspended and new segments
> are not accepted (active waiting). There's a single thread which retries
> adding the segment with some delay. If the segment is successfully written,
> the queue will back to the normal operation.
> This way the unavailable HDFS service is not flooded by the requests and
> we're not accepting the segments when we can't persist them.
> The close() method finishes the recovery mode - in this case, some of the
> awaiting segments won't be persisted.
> h5. Consistency
> The asynchronous mode isn't as reliable as the standard, synchronous case.
> Following cases are possible:
> * TarWriter#writeEntry() returns successfully, but the segments are not
> persisted.
> * TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
> S3 are persisted, but the S1 is not.
> On the other hand:
> * If the TarWriter#flush() returns successfully, it means that all the
> accepted segments has been
[jira] [Updated] (OAK-6921) Support pluggable segment storage
[
https://issues.apache.org/jira/browse/OAK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6921:
---
Attachment: (was: OAK-6921.patch)
> Support pluggable segment storage
> -
>
> Key: OAK-6921
> URL: https://issues.apache.org/jira/browse/OAK-6921
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
> Fix For: 1.9.0
>
> Attachments: OAK-6921.patch
>
>
> h3. Rationale
> segment-tar, as names suggest, stores the segments in a bunch of tar
> archives, inside the {{segmentstore}} directory on the local file system. For
> some cases, especially in the cloud deployments, it may be interesting to
> store the segments outside the local FS - the remote storage such as Amazon
> S3, Azure Blob Storage or HDFS may be cheaper than a mounted disk, more
> scalable, easier for the provisioning, etc.
> h3. Current state
> There are 3 classes responsible for handling tar files in the segment-tar:
> TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}}
> directory, scans it for the .tars and for each one creates a TarReader. It
> also creates a single TarWriter object, used to write (and also read) the
> most recent tar file.
> The TarWriter appends segments to the latest tar file and also serializes the
> auxiliary indexes: segment index, binary references index and the segment
> graph. It also takes of synchronization, as we're dealing with a mutable data
> structure - tar file opened in the append mode.
> The TarReader not only reads the segments from the tar file, but is also
> responsible for the revision GC (mark & sweep methods) and recovering data
> from files which hasn't been closed cleanly (eg. have no index).
> h3. New abstraction layer
> In order to store segments not in the tar files, but somewhere else, it'd be
> possible to create own implementation of the TarFiles, TarWriter and
> TarReader. However, such implementation would duplicate a lot of code, not
> strictly related to the persistence - mark(), sweep(), synchronization, etc.
> Rather than that, the attached patch presents a different approach: a new
> layer of abstraction is injected into TarFiles, TarWriter and TarReader - it
> only takes care of the segments persistence and knows nothing about the
> synchronization, GC, etc. - leaving it to the upper layer.
> The new abstraction layer is modelled using 3 new classes:
> SegmentArchiveManager, SegmentArchiveReader and SegmentArchiveWriter. They
> are strictly related to the existing Tar* classes and used by them.
> SegmentArchiveManager provides a bunch of file system-style methods, like
> open(), create(), delete(), exists(), etc. The open() and create() returns
> instances of the SAReader and SAWriter.
> SegmentArchiveReader, despite from reading segments, can also load and parse
> the index, graph and binary references. The logic responsible for parsing
> these structures has been already extracted, so it doesn't need to be
> duplicated in the SAReader implementations. Also, SAReader needs to be aware
> about the index, since it contains the segment offsets.
> The SAWriter class allows to write and read the segments and also store the
> indexes. It isn't thread safe - it assumes that the synchronization is
> already done on the higher layers.
> In the patch, I've moved the tar implementation to the new classes:
> SegmentTarManager, SegmentTarReader and SegmentTarWriter.
> h3. TODO
> * The names and package locations for all the affected classes are subjects
> to change - after applying the patch the TarFiles doesn't deal with the .tar
> files anymore, similarly the TarReader and TarWriter delegates the low-level
> file access duties to the SegmentArchiveReader and Writer. I didn't want to
> change the names yet, to make it easier to understand and rebase the patch
> with the trunk changes.
> * Add JUnit documentation to the new interfaces.
> * SegmentNodeStoreService should allow to get the SegmentArchiveManager
> service from the OSGi (so the implementations can be added in other bundles).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6921) Support pluggable segment storage
[
https://issues.apache.org/jira/browse/OAK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6921:
---
Attachment: OAK-6921.patch
> Support pluggable segment storage
> -
>
> Key: OAK-6921
> URL: https://issues.apache.org/jira/browse/OAK-6921
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
> Fix For: 1.9.0
>
> Attachments: OAK-6921.patch
>
>
> h3. Rationale
> segment-tar, as names suggest, stores the segments in a bunch of tar
> archives, inside the {{segmentstore}} directory on the local file system. For
> some cases, especially in the cloud deployments, it may be interesting to
> store the segments outside the local FS - the remote storage such as Amazon
> S3, Azure Blob Storage or HDFS may be cheaper than a mounted disk, more
> scalable, easier for the provisioning, etc.
> h3. Current state
> There are 3 classes responsible for handling tar files in the segment-tar:
> TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}}
> directory, scans it for the .tars and for each one creates a TarReader. It
> also creates a single TarWriter object, used to write (and also read) the
> most recent tar file.
> The TarWriter appends segments to the latest tar file and also serializes the
> auxiliary indexes: segment index, binary references index and the segment
> graph. It also takes of synchronization, as we're dealing with a mutable data
> structure - tar file opened in the append mode.
> The TarReader not only reads the segments from the tar file, but is also
> responsible for the revision GC (mark & sweep methods) and recovering data
> from files which hasn't been closed cleanly (eg. have no index).
> h3. New abstraction layer
> In order to store segments not in the tar files, but somewhere else, it'd be
> possible to create own implementation of the TarFiles, TarWriter and
> TarReader. However, such implementation would duplicate a lot of code, not
> strictly related to the persistence - mark(), sweep(), synchronization, etc.
> Rather than that, the attached patch presents a different approach: a new
> layer of abstraction is injected into TarFiles, TarWriter and TarReader - it
> only takes care of the segments persistence and knows nothing about the
> synchronization, GC, etc. - leaving it to the upper layer.
> The new abstraction layer is modelled using 3 new classes:
> SegmentArchiveManager, SegmentArchiveReader and SegmentArchiveWriter. They
> are strictly related to the existing Tar* classes and used by them.
> SegmentArchiveManager provides a bunch of file system-style methods, like
> open(), create(), delete(), exists(), etc. The open() and create() returns
> instances of the SAReader and SAWriter.
> SegmentArchiveReader, despite from reading segments, can also load and parse
> the index, graph and binary references. The logic responsible for parsing
> these structures has been already extracted, so it doesn't need to be
> duplicated in the SAReader implementations. Also, SAReader needs to be aware
> about the index, since it contains the segment offsets.
> The SAWriter class allows to write and read the segments and also store the
> indexes. It isn't thread safe - it assumes that the synchronization is
> already done on the higher layers.
> In the patch, I've moved the tar implementation to the new classes:
> SegmentTarManager, SegmentTarReader and SegmentTarWriter.
> h3. TODO
> * The names and package locations for all the affected classes are subjects
> to change - after applying the patch the TarFiles doesn't deal with the .tar
> files anymore, similarly the TarReader and TarWriter delegates the low-level
> file access duties to the SegmentArchiveReader and Writer. I didn't want to
> change the names yet, to make it easier to understand and rebase the patch
> with the trunk changes.
> * Add JUnit documentation to the new interfaces.
> * SegmentNodeStoreService should allow to get the SegmentArchiveManager
> service from the OSGi (so the implementations can be added in other bundles).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: OAK-6922.patch
> HDFS support for the segment-tar
>
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
> Fix For: 1.9.0
>
> Attachments: OAK-6922.patch
>
>
> A HDFS implementation of the segment storage, based on the OAK-6921 work.
> h3. HDFS
> HDFS is a distributed, network file system. The most popular implementation
> is Apache Hadoop, but the HDFS is also available in the Amazon AWS and
> Microsoft Azure clouds. Despite being a file system, it requires a custom API
> to be used - it's not similar to NFS or CIFS which can be simply mounted
> locally.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> $ hdfs dfs -ls /oak/data0a.tar
> Found 517 items
> -rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
> /oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
> -rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
> /oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
> -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
> /oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
> -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
> /oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
> (...)
> -rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.brf
> -rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.gph
> -rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
> /oak/data0a.tar/data0a.tar.idx
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of HDFS each write involves a network latency. That's
> why the SegmentWriteQueue was introduced. The segments are added to the
> blocking deque, which is served by a number of the consumer threads, writing
> the segments to HDFS. There's also a map UUID->Segment, which allows to
> return the segments in case they are requested by the readSegment() method
> before they are actually persisted. Segments are removed from the map only
> after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. TODO: queue recovery mode
> We need to handle the write failures in a better way: if the HDFS write()
> operation fails, we should re-add the segment to the queue and switch to an
> "recovery mode". In this mode, all the threads are suspended and new segments
> are not accepted (active waiting). There's a single thread which retries
> adding the segment with some delay. If the segment is successfully written,
> the queue will back to the normal operation.
> This way the unavailable HDFS service is not flooded by the requests and
> we're not accepting the segments when we can't persist them.
> The close() method finishes the recovery mode - in this case, some of the
> awaiting segments won't be persisted.
> h5. Consistency
> The asynchronous mode isn't as reliable as the standard, synchronous case.
> Following cases are possible:
> * TarWriter#writeEntry() returns successfully, but the segments are not
> persisted.
> * TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
> S3 are persisted, but the S1 is not.
> On the other hand:
> * If the TarWriter#flush() returns successfully, it means that all the
> accepted segments has been persisted.
> This
[jira] [Updated] (OAK-6921) Support pluggable segment storage
[
https://issues.apache.org/jira/browse/OAK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6921:
---
Description:
h3. Rationale
segment-tar, as names suggest, stores the segments in a bunch of tar archives,
inside the {{segmentstore}} directory on the local file system. For some cases,
especially in the cloud deployments, it may be interesting to store the
segments outside the local FS - the remote storage such as Amazon S3, Azure
Blob Storage or HDFS may be cheaper than a mounted disk, more scalable, easier
for the provisioning, etc.
h3. Current state
There are 3 classes responsible for handling tar files in the segment-tar:
TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}}
directory, scans it for the .tars and for each one creates a TarReader. It also
creates a single TarWriter object, used to write (and also read) the most
recent tar file.
The TarWriter appends segments to the latest tar file and also serializes the
auxiliary indexes: segment index, binary references index and the segment
graph. It also takes of synchronization, as we're dealing with a mutable data
structure - tar file opened in the append mode.
The TarReader not only reads the segments from the tar file, but is also
responsible for the revision GC (mark & sweep methods) and recovering data from
files which hasn't been closed cleanly (eg. have no index).
h3. New abstraction layer
In order to store segments not in the tar files, but somewhere else, it'd be
possible to create own implementation of the TarFiles, TarWriter and TarReader.
However, such implementation would duplicate a lot of code, not strictly
related to the persistence - mark(), sweep(), synchronization, etc. Rather than
that, the attached patch presents a different approach: a new layer of
abstraction is injected into TarFiles, TarWriter and TarReader - it only takes
care of the segments persistence and knows nothing about the synchronization,
GC, etc. - leaving it to the upper layer.
The new abstraction layer is modelled using 3 new classes:
SegmentArchiveManager, SegmentArchiveReader and SegmentArchiveWriter. They are
strictly related to the existing Tar* classes and used by them.
SegmentArchiveManager provides a bunch of file system-style methods, like
open(), create(), delete(), exists(), etc. The open() and create() returns
instances of the SAReader and SAWriter.
SegmentArchiveReader, despite from reading segments, can also load and parse
the index, graph and binary references. The logic responsible for parsing these
structures has been already extracted, so it doesn't need to be duplicated in
the SAReader implementations. Also, SAReader needs to be aware about the index,
since it contains the segment offsets.
The SAWriter class allows to write and read the segments and also store the
indexes. It isn't thread safe - it assumes that the synchronization is already
done on the higher layers.
In the patch, I've moved the tar implementation to the new classes:
SegmentTarManager, SegmentTarReader and SegmentTarWriter.
h3. TODO
* The names and package locations for all the affected classes are subjects to
change - after applying the patch the TarFiles doesn't deal with the .tar files
anymore, similarly the TarReader and TarWriter delegates the low-level file
access duties to the SegmentArchiveReader and Writer. I didn't want to change
the names yet, to make it easier to understand and rebase the patch with the
trunk changes.
* SegmentNodeStoreService should allow to get the SegmentArchiveManager service
from the OSGi (so the implementations can be added in other bundles).
was:
h3. Rationale
segment-tar, as names suggest, stores the segments in a bunch of tar archives,
inside the {{segmentstore}} directory on the local file system. For some cases,
especially in the cloud deployments, it may be interesting to store the
segments outside the local FS - the remote storage such as Amazon S3, Azure
Blob Storage or HDFS may be cheaper than a mounted disk, more scalable, easier
for the provisioning, etc.
h3. Current state
There are 3 classes responsible for handling tar files in the segment-tar:
TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}}
directory, scans it for the .tars and for each one creates a TarReader. It also
creates a single TarWriter object, used to write (and also read) the most
recent tar file.
The TarWriter appends segments to the latest tar file and also serializes the
auxiliary indexes: segment index, binary references index and the segment
graph. It also takes of synchronization, as we're dealing with a mutable data
structure - tar file opened in the append mode.
The TarReader not only reads the segments from the tar file, but is also
responsible for the revision GC (mark & sweep methods) and recovering data from
files which hasn't been closed
[jira] [Updated] (OAK-6867) Enable index hints on UUID lookup query
[
https://issues.apache.org/jira/browse/OAK-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-6867:
Sprint: L15
> Enable index hints on UUID lookup query
> ---
>
> Key: OAK-6867
> URL: https://issues.apache.org/jira/browse/OAK-6867
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Alex Deparvu
>Assignee: Thomas Mueller
> Fix For: 1.8
>
>
> For the UUID lookup query I enabled the (recently added) query hints support
> where you can specify the index you want to be used and I got an interesting
> speedup in the benchmarks:
> {noformat}
> Hint disabled, lookup by query
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[query]
> Oak-Segment-Tar1 5 5 6 7 27
> 9837
> Hint enabled, lookup by query
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[query]
> Oak-Segment-Tar1 5 5 6 7 13
> 10128
> Hint enabled, lookup by getNode
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[Session#getNodeByIdentifier]
> Oak-Segment-Tar1 5 6 6 7 13
> 9447
> Hint disabled, lookup by getNode
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[Session#getNodeByIdentifier]
> Oak-Segment-Tar1 3 3 3 4 8
> 17504
> {noformat}
> It seems the cost shortcut is reducing the overall time, but cannot say for
> sure where this change comes from.
> The only change I did was to add {{option(index name uuid)}} to this query [0]
> [~tmueller], [~chetanm] what do you think?
> [0]
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/identifier/IdentifierManager.java#L356
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6867) Enable index hints on UUID lookup query
[
https://issues.apache.org/jira/browse/OAK-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Deparvu updated OAK-6867:
--
Fix Version/s: 1.8
> Enable index hints on UUID lookup query
> ---
>
> Key: OAK-6867
> URL: https://issues.apache.org/jira/browse/OAK-6867
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Alex Deparvu
>Assignee: Thomas Mueller
> Fix For: 1.8
>
>
> For the UUID lookup query I enabled the (recently added) query hints support
> where you can specify the index you want to be used and I got an interesting
> speedup in the benchmarks:
> {noformat}
> Hint disabled, lookup by query
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[query]
> Oak-Segment-Tar1 5 5 6 7 27
> 9837
> Hint enabled, lookup by query
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[query]
> Oak-Segment-Tar1 5 5 6 7 13
> 10128
> Hint enabled, lookup by getNode
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[Session#getNodeByIdentifier]
> Oak-Segment-Tar1 5 6 6 7 13
> 9447
> Hint disabled, lookup by getNode
> # UUIDLookupTest C min 10% 50% 90% max
> N
> No of indexes (60) 60, Lookup by (lookupByQuery)[Session#getNodeByIdentifier]
> Oak-Segment-Tar1 3 3 3 4 8
> 17504
> {noformat}
> It seems the cost shortcut is reducing the overall time, but cannot say for
> sure where this change comes from.
> The only change I did was to add {{option(index name uuid)}} to this query [0]
> [~tmueller], [~chetanm] what do you think?
> [0]
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/identifier/IdentifierManager.java#L356
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6917) Configuration presets for DocumentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245560#comment-16245560
]
Marcel Reutegger commented on OAK-6917:
---
Nope, that doesn't work:
{noformat}
org.apache.jackrabbit.oak-store-document bundle
org.apache.jackrabbit.oak-store-document:1.8.0.SNAPSHOT
(107)[org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreService(213)]
: Field preset in class class
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreService has
unsupported type org.apache.jackrabbit.oak.plugins.document.Configuration
{noformat}
Going through the indirection of another service with an optional configuration
works though.
> Configuration presets for DocumentNodeStoreService
> --
>
> Key: OAK-6917
> URL: https://issues.apache.org/jira/browse/OAK-6917
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch
>
>
> When Oak is deployed in an OSGi container, applications usually want to ship
> a default configuration which is different from the defaults present in Oak.
> E.g. an application may want to use a default cache size of 1G for the
> DocumentNodeStoreService instead of the default 256M. Now if a user of the
> application provides a custom configuration and does not specify the cache
> size, the value for this configuration will flip back to the Oak default of
> 256M.
> There should be a way to configure presets for the application that are
> different from the Oak defaults and then allow a user to customize the
> configuration while still respecting the presets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6917) Configuration presets for DocumentNodeStoreService
[ https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245538#comment-16245538 ] Marcel Reutegger commented on OAK-6917: --- Interesting... Thanks for the hint. I didn't know you could reference the configuration of another component. I'll give this a try. > Configuration presets for DocumentNodeStoreService > -- > > Key: OAK-6917 > URL: https://issues.apache.org/jira/browse/OAK-6917 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.8 > > Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch > > > When Oak is deployed in an OSGi container, applications usually want to ship > a default configuration which is different from the defaults present in Oak. > E.g. an application may want to use a default cache size of 1G for the > DocumentNodeStoreService instead of the default 256M. Now if a user of the > application provides a custom configuration and does not specify the cache > size, the value for this configuration will flip back to the Oak default of > 256M. > There should be a way to configure presets for the application that are > different from the Oak defaults and then allow a user to customize the > configuration while still respecting the presets. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6915) Minimize the amount of uncached segment reads
[
https://issues.apache.org/jira/browse/OAK-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari updated OAK-6915:
Attachment: OAK-6915-diagnostics-02.patch
I'm not convinced about the third version of the patch and the usefulness of
{{SegmentId#unloaded}}.
In order to gather some data, I improved the diagnostics patch. The patch now
computes how many segments are written, so to have a feeling about the amount
of unique segments the {{FileStore}} has to work with. Additionally, every
statistics has been split for data and bulk segments. Since {{SegmentCache}}
behaves differently for data and bulk segments, it makes sense to gather
separate numbers.
I run {{StandbyTestIT#testSyncLoop}} with the diagnostic patch in place and
some variations.
|trunk|{noformat}TarMK data segment ID allocations: 54
TarMK bulk segment ID allocations: 4
TarMK uncached data segment reads: 99854
TarMK uncached bulk segment reads: 2
TarMK written data segments..: 30
TarMK written bulk segments..: 2
{noformat}|
|OAK-6915.patch|{noformat}TarMK data segment ID allocations: 56
TarMK bulk segment ID allocations: 4
TarMK uncached data segment reads: 101649
TarMK uncached bulk segment reads: 2
TarMK written data segments..: 32
TarMK written bulk segments..: 2{noformat}|
|OAK-6915-02.patch|{noformat}TarMK data segment ID allocations: 38
TarMK bulk segment ID allocations: 2
TarMK uncached data segment reads: 33
TarMK uncached bulk segment reads: 2
TarMK written data segments..: 29
TarMK written bulk segments..: 2{noformat}|
Even if we ignore the number of uncached data segment reads, the ratio between
data segment ID allocations and number of written data segments shows something
important: segment IDs are not as perfectly internalized as we thought. This
has two important consequences.
First, no matter if we call {{SegmentId#unloaded}}, there will probably be
another {{SegmentId}} out there with the same MSB/LSB that contains a reference
to the segment. Actually, as speculated in OAK-6919, {{SegmentCache}} itself
might be the culprit of this behaviour.
Second, {{SegmentCache}} should not depend on {{SegmentId}} to be perfectly
internalized. It is very easy to break this design assumption, and the
consequences of it are usually disastrous. It is, in my opinion, better to
implement a cache that doesn't work with this assumption in mind.
> Minimize the amount of uncached segment reads
> -
>
> Key: OAK-6915
> URL: https://issues.apache.org/jira/browse/OAK-6915
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
> Fix For: 1.8, 1.7.12
>
> Attachments: OAK-6915-01.patch, OAK-6915-02.patch,
> OAK-6915-diagnostics-02.patch, OAK-6915-diagnostics.patch, OAK-6915.patch
>
>
> The current implementation of {{SegmentCache}} should make better use of the
> underlying Guava cache by relying on the cached segments instead of
> unconditionally performing an uncached segment read via the
> {{Callable}} passed to {{SegmentCache#getSegment}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6917) Configuration presets for DocumentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-6917:
Attachment: OAK-6917-alternative-approach.patch
[~mreutegg] I attached a patch with a sketch for an alternative approach, YMMV.
It is limited to the OSGi aspect, but lacks merging the "preset" (or default)
configuration with the config for {{DocumentNodeStoreService}}. Check it out,
maybe you find it useful.
Basically, I register an additional service, that references the same
{{Configuration}} annotation. That way, it gets all the defaults set in the
annotation, but these can be overridden by config admin (config policy is
optional though). I created the (mostly empty) inner class {{Defaults}} to
achieve this. The {{DocumentNodeStoreService}} then references the
{{DNSS.Defaults}} service, but instead of having the service instance injected,
we ask to get the {{Configuration}} annotation injected (not 100% sure this
works, if not, try with {{Map}}, which I have done before).
I didn't test the patch, but quickly looked at the generated XML files, and
they look as intended (at first glance at least). Note that I deliberately
didn't get metatype information generated for the {{DNSS.Defaults}} service, as
I understood that it should be a "private" config that should only be set by
product vendors/distributors.
> Configuration presets for DocumentNodeStoreService
> --
>
> Key: OAK-6917
> URL: https://issues.apache.org/jira/browse/OAK-6917
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch
>
>
> When Oak is deployed in an OSGi container, applications usually want to ship
> a default configuration which is different from the defaults present in Oak.
> E.g. an application may want to use a default cache size of 1G for the
> DocumentNodeStoreService instead of the default 256M. Now if a user of the
> application provides a custom configuration and does not specify the cache
> size, the value for this configuration will flip back to the Oak default of
> 256M.
> There should be a way to configure presets for the application that are
> different from the Oak defaults and then allow a user to customize the
> configuration while still respecting the presets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6542) java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir
[
https://issues.apache.org/jira/browse/OAK-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245351#comment-16245351
]
Francesco Mari commented on OAK-6542:
-
[~olamy], the fix for this issue is to put {{metrics-core}} on the classpath,
like you already do for every other {{provided}} dependency declared by
{{oak-segment-tar}}.
> java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir
> --
>
> Key: OAK-6542
> URL: https://issues.apache.org/jira/browse/OAK-6542
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.7.5
>Reporter: Olivier Lamy (*$^¨%`£)
>Assignee: Francesco Mari
> Fix For: 1.8, 1.7.12
>
>
> Upgrading to last 1.7.5.
> I get this exception
> java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.(SegmentNodeStore.java:166)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.(SegmentNodeStore.java:63)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$SegmentNodeStoreBuilder.build(SegmentNodeStore.java:121)
> Looking at the pom the dependency has a scope provided
> (http://repo.maven.apache.org/maven2/org/apache/jackrabbit/oak-segment-tar/1.7.5/oak-segment-tar-1.7.5.pom)
> IMHO it's a wrong dependency scope at it's definitely needed as there is no
> usage of reflection to avoid loading of the classes
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or CIFS which can be simply mounted locally.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
$ hdfs dfs -ls /oak/data0a.tar
Found 517 items
-rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
/oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
-rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
/oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
(...)
-rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.brf
-rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.gph
-rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.idx
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of HDFS each write involves a network latency. That's why
the SegmentWriteQueue was introduced. The segments are added to the blocking
deque, which is served by a number of the consumer threads, writing the
segments to HDFS. There's also a map UUID->Segment, which allows to return the
segments in case they are requested by the readSegment() method before they are
actually persisted. Segments are removed from the map only after a successful
write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h5. TODO: queue recovery mode
We need to handle the write failures in a better way: if the HDFS write()
operation fails, we should re-add the segment to the queue and switch to an
"recovery mode". In this mode, all the threads are suspended and new segments
are not accepted (active waiting). There's a single thread which retries adding
the segment with some delay. If the segment is successfully written, the queue
will back to the normal operation.
This way the unavailable HDFS service is not flooded by the requests and we're
not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h5. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments are not
persisted.
* TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
S3 are persisted, but the S1 is not.
On the other hand:
* If the TarWriter#flush() returns successfully, it means that all the accepted
segments has been persisted.
This may lead to data inconsistency - especially the second case, where we lose
a middle segment. The impact is still to be discussed.
h3. TODO
* implement the queue recovery mode, as above,
* test the queue,
* move the implementation to its own bundle (requires OSGi support for
SegmentArchvieManager).
was:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or CIFS which can be simply mounted locally.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
$ hdfs dfs -ls /oak/data0a.tar
Found 517 items
-rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
/oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
-rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
/oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
(...)
-rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.brf
-rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.gph
-rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.idx
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of HDFS each write involves a network latency. That's why
the SegmentWriteQueue was introduced. The segments are added to the blocking
deque, which is served by a number of the consumer threads, writing the
segments to HDFS. There's also a map UUID->Segment, which allows to return the
segments in case they are requested by the readSegment() method before they are
actually persisted. Segments are removed from the map only after a successful
write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h5. TODO: queue recovery mode
We need to handle the write failures in a better way: if the HDFS write()
operation fails, we should re-add the segment to the queue and switch to an
"recovery mode". In this mode, all the threads are suspended and new segments
are not accepted (active waiting). There's a single thread which retries adding
the segment with some delay. If the segment is successfully written, the queue
will back to the normal operation.
This way the unavailable HDFS service is not flooded by the requests and we're
not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h5. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments are not
persisted.
* TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
S3 are persisted, but the S1 is not.
On the other hand:
* If the TarWriter#flush() returns successfully, it means that all the accepted
segments has been persisted.
This may lead to data inconsistency - especially the second case, where we lose
a middle segment. The impact is till to be discussed.
h3. TODO
* implement the queue recovery mode, as above,
* test the queue,
* move the implementation to its own bundle (requires OSGi support for
SegmentArchvieManager).
was:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or CIFS which can be simply mounted locally.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
$ hdfs dfs -ls /oak/data0a.tar
Found 517 items
-rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
/oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
-rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
/oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
(...)
-rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.brf
-rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.gph
-rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.idx
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of HDFS each write involves a network latency. That's why
the SegmentWriteQueue was introduced. The segments are added to the blocking
deque, which is served by a number of the consumer threads, writing the
segments to HDFS. There's also a map UUID->Segment, which allows to return the
segments in case they are requested by the readSegment() method before they are
actually persisted. Segments are removed from the map only after a successful
write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h5. TODO: queue recovery mode
We need to handle the write failures in a better way: if the HDFS write()
operation fails, we should re-add the segment to the queue and switch to an
"recovery mode". In this mode, all the threads are suspended and new segments
are not accepted (active waiting). There's a single thread which retries adding
the segment with some delay. If the segment is successfully written, the queue
will back to the normal operation.
This way the unavailable HDFS service is not flooded by the requests and we're
not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h5. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments are not
persisted.
* TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
S3 are persisted, but the S1 is not.
On the other hand:
* If the TarWriter#flush() returns successfully, it means that all the accepted
segments has been persisted.
This may lead to data inconsistency - especially the second case, where we lose
a middle segment. The impact is till to be discussed.
h3. TODO
* implement the queue recovery mode, as above,
* move the implementation to its own bundle (requires OSGi support for
SegmentArchvieManager).
was:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure
[jira] [Updated] (OAK-6922) HDFS support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure clouds. Despite being a file system, it requires a custom API to be used
- it's not similar to NFS or CIFS which can be simply mounted locally.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
$ hdfs dfs -ls /oak/data0a.tar
Found 517 items
-rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24
/oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40
-rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24
/oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb
-rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24
/oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a
(...)
-rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.brf
-rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.gph
-rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32
/oak/data0a.tar/data0a.tar.idx
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of HDFS each write involves a network latency. That's why
the SegmentWriteQueue was introduced. The segments are added to the blocking
deque, which is served by a number of the consumer threads, writing the
segments to HDFS. There's also a map UUID->Segment, which allows to return the
segments in case they are requested by the readSegment() method before they are
actually persisted. Segments are removed from the map only after a successful
write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h4. TODO: queue recovery mode
We need to handle the write failures in a better way: if the HDFS write()
operation fails, we should re-add the segment to the queue and switch to an
"recovery mode". In this mode, all the threads are suspended and new segments
are not accepted (active waiting). There's a single thread which retries adding
the segment with some delay. If the segment is successfully written, the queue
will back to the normal operation.
This way the unavailable HDFS service is not flooded by the requests and we're
not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h4. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments are not
persisted.
* TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and
S3 are persisted, but the S1 is not.
On the other hand:
* If the TarWriter#flush() returns successfully, it means that all the accepted
segments has been persisted.
This may lead to data inconsistency - especially the second case, where we lose
a middle segment. The impact is till to be discussed.
h3. TODO
* implement the queue recovery mode, as above,
* move the implementation to its own bundle (requires OSGi support for
SegmentArchvieManager).
was:
A HDFS implementation of the segment storage, based on the OAK-6921 work.
h3. HDFS
HDFS is a distributed, network file system. The most popular implementation is
Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft
Azure
