[jira] [Commented] (OAK-6544) Enable active blob deletion feature by default
[ https://issues.apache.org/jira/browse/OAK-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122883#comment-16122883 ] Amit Jain commented on OAK-6544: This has to be linked to the synchronization interval for the BlobTracker (The thing which caches the blob ids locally), where its frequency has to be less than the frequency here so that it can synchronize the deletes before the full scale GC takes place. > Enable active blob deletion feature by default > -- > > Key: OAK-6544 > URL: https://issues.apache.org/jira/browse/OAK-6544 > Project: Jackrabbit Oak > Issue Type: Task > Components: lucene >Affects Versions: 1.7.1 >Reporter: Vikas Saurabh >Assignee: Vikas Saurabh > Fix For: 1.8 > > > OAK-2808 allows for active deletion of blobs created by lucene indexing. > Currently, the feature is disabled by default. This task is to track enabling > it by default before doing stable release of the feature. > About the default enabled value: I think default interval to purge blobs can > be 6 hours be default. (Note, the purge logic, anyway won't purge blobs which > were deleted in recent past... 'recent'=24hours \[configurable via jvm > param]). > /cc [~chetanm] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6545) Tooling to serialize NodeState as json along with blobs
[ https://issues.apache.org/jira/browse/OAK-6545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-6545: - Summary: Tooling to serialize NodeState as json along with blobs (was: Tooling to serilize NodeState as json along with blobs) > Tooling to serialize NodeState as json along with blobs > --- > > Key: OAK-6545 > URL: https://issues.apache.org/jira/browse/OAK-6545 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > > For debugging certain cases like OAK-6525 we need a way to analyze the hidden > NodeState structure used by indexes. To simplify the effort I would like to > add some tooling to oak-run which allows dumping the NodeState and its > children for certain path along with the blob contents -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6546) JsonSerializer should taken an instance of JsopWriter
Chetan Mehrotra created OAK-6546:
Summary: JsonSerializer should taken an instance of JsopWriter
Key: OAK-6546
URL: https://issues.apache.org/jira/browse/OAK-6546
Project: Jackrabbit Oak
Issue Type: Technical task
Components: store-spi
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.8
{{JsonSerializer}} currently taken an instance of {{JsopBuilder}}. The internal
logic can work with {{JsopWriter}}. So change the logic to work with
{{JsopWriter}}
This would allow passing in a streaming writer which does not accumulate the
json content in memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6545) Tooling to serilize NodeState as json along with blobs
Chetan Mehrotra created OAK-6545: Summary: Tooling to serilize NodeState as json along with blobs Key: OAK-6545 URL: https://issues.apache.org/jira/browse/OAK-6545 Project: Jackrabbit Oak Issue Type: New Feature Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8 For debugging certain cases like OAK-6525 we need a way to analyze the hidden NodeState structure used by indexes. To simplify the effort I would like to add some tooling to oak-run which allows dumping the NodeState and its children for certain path along with the blob contents -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6544) Enable active blob deletion feature by default
Vikas Saurabh created OAK-6544: -- Summary: Enable active blob deletion feature by default Key: OAK-6544 URL: https://issues.apache.org/jira/browse/OAK-6544 Project: Jackrabbit Oak Issue Type: Task Components: lucene Affects Versions: 1.7.1 Reporter: Vikas Saurabh Assignee: Vikas Saurabh Fix For: 1.8 OAK-2808 allows for active deletion of blobs created by lucene indexing. Currently, the feature is disabled by default. This task is to track enabling it by default before doing stable release of the feature. About the default enabled value: I think default interval to purge blobs can be 6 hours be default. (Note, the purge logic, anyway won't purge blobs which were deleted in recent past... 'recent'=24hours \[configurable via jvm param]). /cc [~chetanm] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6543) NodeCounter: JMX description
[ https://issues.apache.org/jira/browse/OAK-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller resolved OAK-6543. - Resolution: Fixed Fix Version/s: 1.7.6 http://svn.apache.org/r1804665 > NodeCounter: JMX description > > > Key: OAK-6543 > URL: https://issues.apache.org/jira/browse/OAK-6543 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: query >Reporter: Thomas Mueller >Assignee: Thomas Mueller >Priority: Minor > Fix For: 1.8, 1.7.6 > > > The NodeCounter bean doesn't have JMX descriptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6543) NodeCounter: JMX description
[ https://issues.apache.org/jira/browse/OAK-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-6543: Component/s: query > NodeCounter: JMX description > > > Key: OAK-6543 > URL: https://issues.apache.org/jira/browse/OAK-6543 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: query >Reporter: Thomas Mueller >Assignee: Thomas Mueller >Priority: Minor > Fix For: 1.8 > > > The NodeCounter bean doesn't have JMX descriptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6543) NodeCounter: JMX description
Thomas Mueller created OAK-6543: --- Summary: NodeCounter: JMX description Key: OAK-6543 URL: https://issues.apache.org/jira/browse/OAK-6543 Project: Jackrabbit Oak Issue Type: Improvement Reporter: Thomas Mueller Assignee: Thomas Mueller Priority: Minor Fix For: 1.8 The NodeCounter bean doesn't have JMX descriptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6542) java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir
Olivier Lamy (*$^¨%`£) created OAK-6542: --- Summary: java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir Key: OAK-6542 URL: https://issues.apache.org/jira/browse/OAK-6542 Project: Jackrabbit Oak Issue Type: Bug Components: segment-tar Affects Versions: 1.7.5 Reporter: Olivier Lamy (*$^¨%`£) Upgrading to last 1.7.5. I get this exception java.lang.NoClassDefFoundError: com/codahale/metrics/Reservoir at org.apache.jackrabbit.oak.segment.SegmentNodeStore.(SegmentNodeStore.java:166) at org.apache.jackrabbit.oak.segment.SegmentNodeStore.(SegmentNodeStore.java:63) at org.apache.jackrabbit.oak.segment.SegmentNodeStore$SegmentNodeStoreBuilder.build(SegmentNodeStore.java:121) Looking at the pom the dependency has a scope provided (http://repo.maven.apache.org/maven2/org/apache/jackrabbit/oak-segment-tar/1.7.5/oak-segment-tar-1.7.5.pom) IMHO it's a wrong dependency at it's definitely needed as there is no usage of reflection to avoid loading of the classes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6540) Session.hasAccess(...) should reflect read-only status of mounts
[
https://issues.apache.org/jira/browse/OAK-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121504#comment-16121504
]
Robert Munteanu commented on OAK-6540:
--
Thanks for the pointer [~anchela], should have kept reading that page :-) I'll
report a separate issue for Session.hasCapability and look into how that would
work
> Session.hasAccess(...) should reflect read-only status of mounts
>
>
> Key: OAK-6540
> URL: https://issues.apache.org/jira/browse/OAK-6540
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: composite, security
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.6
>
>
> When a mount is set in read-only mode callers that check
> {{Session.hasPermission("set_property", ...)}} or
> {{Session.hasPermission("add_node", ...)}} for mounted paths will believe
> that they are able to write under those paths. For a composite setup with a
> read-only mount this should (IMO) reflect that callers are not able to write,
> taking into account the mount information on top of the ACEs.
> [~anchela], [~stillalex] - WDYT?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6534) Compute indexPaths from index definitions json
[
https://issues.apache.org/jira/browse/OAK-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-6534:
Description:
Currently while adding/updating indexes via {{\-\-index-definitions-file}}
(OAK-6471) the index paths are always determined by {{\-\-index-paths}} option.
If there are more index definitions present in the json file then those would
be ignored.
To avoid confusion following approach should be implemented
* If {{\-\-index-paths}} is specified then use that
* If not and {{\-\-index-definitions-file}} is provided then compute index
paths from that
* If both are specified then {{\-\-index-paths}} then merge as user may want to
reindex few indexes and also update few others
was:
Currently while adding/updating indexes via {{--index-definitions-file}}
(OAK-6471) the index paths are always determined by {{--index-paths}} option.
If there are more index definitions present in the json file then those would
be ignored.
To avoid confusion following approach should be implemented
* If {{--index-paths}} is specified then use that
* If not and {{--index-definitions-file}} is provided then compute index paths
from that
* If both are specified then {{--index-paths}} then merge as user may want to
reindex few indexes and also update few others
> Compute indexPaths from index definitions json
> --
>
> Key: OAK-6534
> URL: https://issues.apache.org/jira/browse/OAK-6534
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.8, 1.7.6
>
>
> Currently while adding/updating indexes via {{\-\-index-definitions-file}}
> (OAK-6471) the index paths are always determined by {{\-\-index-paths}}
> option. If there are more index definitions present in the json file then
> those would be ignored.
> To avoid confusion following approach should be implemented
> * If {{\-\-index-paths}} is specified then use that
> * If not and {{\-\-index-definitions-file}} is provided then compute index
> paths from that
> * If both are specified then {{\-\-index-paths}} then merge as user may want
> to reindex few indexes and also update few others
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6500) NRTIndex leaks file handles due to unclosed IndexReader
[
https://issues.apache.org/jira/browse/OAK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-6500:
-
Labels: (was: candidate_oak_1_6)
Fix Version/s: 1.6.4
Merged to 1.6 with 1804664
> NRTIndex leaks file handles due to unclosed IndexReader
> ---
>
> Key: OAK-6500
> URL: https://issues.apache.org/jira/browse/OAK-6500
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.6.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Fix For: 1.8, 1.7.6, 1.6.4
>
> Attachments: OAK-6500-approach-a-v1.patch,
> OAK-6500-ref-count-v1.patch, OAK-6500-v1.patch
>
>
> On some setups under stress it has been seen that NRTIndex leaks file handles
> over time.
> Checking with lsof indicates that more than 3 nrt folders per index are being
> used. However per design there can be max 3 and after system is not in use
> max 1 should be present.
> {noformat}
> $ lsof -p 9550 | grep '/nrt' | gawk 'match($0,
> /.*crx-quickstart\/repository\/index\/(.*?)\/\_.*$/, m) { print m[1]; }' |
> sort | uniq
> cqPageLucene-1501065263331/nrt1501065335930
> cqPageLucene-1501065263331/nrt1501065374667
> cqPageLucene-1501065263331/nrt1501065392492
> cqPageLucene-1501065263331/nrt1501065440955
> cqPageLucene-1501065263331/nrt1501065473286
> cqPageLucene-1501065263331/nrt1501065507345
> slingeventJob-1501065263330/nrt1501065356975
> slingeventJob-1501065263330/nrt1501065373229
> slingeventJob-1501065263330/nrt1501065394142
> slingeventJob-1501065263330/nrt1501065440953
> slingeventJob-1501065263330/nrt1501065473282
> slingeventJob-1501065263330/nrt1501065507342
> versionStoreIndex-1501065263332/nrt1501065335925
> versionStoreIndex-1501065263332/nrt1501065366781
> versionStoreIndex-1501065263332/nrt1501065392490
> versionStoreIndex-1501065263332/nrt1501065441232
> versionStoreIndex-1501065263332/nrt1501065473285
> {noformat}
> Further actually checking index folder indicates that those folder are
> actually deleted. So some where the file handle is still referring them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6539) Decrease version export for org.apache.jackrabbit.oak.spi.security.authentication
[
https://issues.apache.org/jira/browse/OAK-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Deparvu resolved OAK-6539.
---
Resolution: Fixed
Fix Version/s: 1.7.6
thanks [~rombert] for the detailed analysis!
decreased the version with http://svn.apache.org/viewvc?rev=1804663=rev
> Decrease version export for
> org.apache.jackrabbit.oak.spi.security.authentication
> -
>
> Key: OAK-6539
> URL: https://issues.apache.org/jira/browse/OAK-6539
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
>Priority: Trivial
> Fix For: 1.7.6
>
>
> There's a warning when building oak-core related to the export version for
> the org.apache.jackrabbit.oak.spi.security.authentication package:
> {noformat}
> [WARNING] org.apache.jackrabbit.oak.spi.security.authentication: Excessive
> version increase; detected 1.3.0, suggested 1.2.0
> {noformat}
> I see no reason to not decrease the version. [~anchela], thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6539) Decrease version export for org.apache.jackrabbit.oak.spi.security.authentication
[
https://issues.apache.org/jira/browse/OAK-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121434#comment-16121434
]
Robert Munteanu commented on OAK-6539:
--
This behaviour stems from the fact that the comparison baseline is 1.6.2, not
the highest released version.
The exported version was increased in
[r1800179|https://svn.apache.org/r1800179], so if the baseline goal would take
into account 1.7.5, there would be an error, requiring increasing the version
to 1.3.1, due to the annotation type changes. So it's probably OK to decrease
the version to 1.2.0, as we (AFAIU) have the policy of managing package version
changes only based on stable releases.
> Decrease version export for
> org.apache.jackrabbit.oak.spi.security.authentication
> -
>
> Key: OAK-6539
> URL: https://issues.apache.org/jira/browse/OAK-6539
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
>Priority: Trivial
>
> There's a warning when building oak-core related to the export version for
> the org.apache.jackrabbit.oak.spi.security.authentication package:
> {noformat}
> [WARNING] org.apache.jackrabbit.oak.spi.security.authentication: Excessive
> version increase; detected 1.3.0, suggested 1.2.0
> {noformat}
> I see no reason to not decrease the version. [~anchela], thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6539) Decrease version export for org.apache.jackrabbit.oak.spi.security.authentication
[
https://issues.apache.org/jira/browse/OAK-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121281#comment-16121281
]
Alex Deparvu commented on OAK-6539:
---
bq. Are there @ProviderType interfaces exposed by this package? If so, I think
it's unsafe to change the version back.
yes I see a few of those in the package, but the tooling seems to disagree with
you, if we trust it to enforce version bumps up why not trust it to decrease
versions?
I guess just waiting for the code to change is also one option, but then the
same clients you mentioned that import the bigger version will again break as
they will not see the changes (new code but no new version export).
I'm open to alternatives, what needs to be done to get rid of the warning?
> Decrease version export for
> org.apache.jackrabbit.oak.spi.security.authentication
> -
>
> Key: OAK-6539
> URL: https://issues.apache.org/jira/browse/OAK-6539
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
>Priority: Trivial
>
> There's a warning when building oak-core related to the export version for
> the org.apache.jackrabbit.oak.spi.security.authentication package:
> {noformat}
> [WARNING] org.apache.jackrabbit.oak.spi.security.authentication: Excessive
> version increase; detected 1.3.0, suggested 1.2.0
> {noformat}
> I see no reason to not decrease the version. [~anchela], thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6523) Tune DocumentNodeStore setup for indexing flow for index command
[
https://issues.apache.org/jira/browse/OAK-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-6523.
--
Resolution: Fixed
Fix Version/s: 1.7.6
Done with 1804642
* Configures persistent cache with "size=4096,binary=0,-nodes,-children"
* Configure cache size to minimum(4GB, 1/2*heap memory). Default heap memory
[varies from setup to
setup|https://stackoverflow.com/questions/4667483/how-is-the-default-java-heap-size-determined]
* Configures the cache segment size to 1 for single threaded access
> Tune DocumentNodeStore setup for indexing flow for index command
>
>
> Key: OAK-6523
> URL: https://issues.apache.org/jira/browse/OAK-6523
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.8, 1.7.6
>
>
> Currently oak-run index command uses default DocumentNodeStore setup for all
> actions. It would be better if we can tune the DocumentNodeStore for specific
> actions.
> For e.g. for read only indexing phase we should configure
> # Persistent cache for previous documents and disable it for nodes and
> children
> {noformat}
> -Doak.documentMK.persCache="temp/cache,size=4096,binary=0,-nodes,-children"
> {noformat}
> # Configure cache size to some percentage of allocated heap. To start with
> say 50%
> Any such tuning should only be done if it is not done explicitly i.e. it
> should be possible to override such settings
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6524) Provide an extension point to customize the NodeStore builders
[ https://issues.apache.org/jira/browse/OAK-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-6524. -- Resolution: Fixed Fix Version/s: 1.7.6 Done with 1804641 > Provide an extension point to customize the NodeStore builders > -- > > Key: OAK-6524 > URL: https://issues.apache.org/jira/browse/OAK-6524 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.6 > > > Some commands may require custom config options of the NodeStore builders. > NodeStoreFixtureProvider should expose some extension point to customize the > builder. > For now we can expose 2 interfaces > * DocumentBuilderCustomizer > * FileStoreBuilderCustomizer > Commands can register there customizers with Whiteboard attached to Option > which would then be used by the fixture implementation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6457) Increment the segment version number
[ https://issues.apache.org/jira/browse/OAK-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121188#comment-16121188 ] Francesco Mari commented on OAK-6457: - As discussed offline with [~mduerig], one problem related to this issue is synthesising the generation number and the compacted flag for old segments, which are associated with only a full generation number. The safest way to achieve this is to synthesise the generation number to be equal to the full generation number, and to assume that the compacted flag is always on in old segments. Not only this is the safest approach when dealing with old segments, but it also works seamlessly with the new approach to full and tail compaction. > Increment the segment version number > > > Key: OAK-6457 > URL: https://issues.apache.org/jira/browse/OAK-6457 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.6 > > > OAK-3349 introduced the tail generation number in the segment header. This > change assigns semantics to a portion of the header that was previously not > used. As such, the segment version should be incremented. Proper checks > should also be put in place to transparently migrate segments from the old to > the new version number. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6528) Implement transparent usage of different binary references index formats
[ https://issues.apache.org/jira/browse/OAK-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Mari resolved OAK-6528. - Resolution: Fixed Fixed at r1804638. > Implement transparent usage of different binary references index formats > > > Key: OAK-6528 > URL: https://issues.apache.org/jira/browse/OAK-6528 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.6 > > > The segment store should be able to read and validate two different format of > binary reference indices, the one in use before the introduction of the tail > generation number and the one after. The detection of the different index > types should be fully transparent and automatic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
