[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870191#comment-13870191
]
Alexander Klimetschek commented on OAK-1133:
Here are some ideas I once had for performance tests wrt observation listeners
(adding here to at least persist the idea):
- real world: many listeners (200+), with multiple parallel writes
- external events
- slow, badly implemented listener (random sleep or many jcr read operations)
- compare in-listener filtering w/ early filtering
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.14
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847620#comment-13847620
]
Michael Dürig commented on OAK-1133:
With http://svn.apache.org/r1550756 the new filter support can be leveraged
through (the Oak specific) {{addEventListener}} method that takes a
{{FilterProvider}} as its arguments. The new {{FilterBuilder}} class offers an
easy way to build arbitrarily complex filters.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.14
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836493#comment-13836493
]
Michael Dürig commented on OAK-1133:
At http://svn.apache.org/r1546982 I added selectors for filtering on
child/parent nodes of the node for which an event is reported.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.13
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833911#comment-13833911
]
Michael Dürig commented on OAK-1133:
At http://svn.apache.org/r1546047 I added a path filter with simple globbing
support
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.13
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13826317#comment-13826317
]
Michael Dürig commented on OAK-1133:
* At http://svn.apache.org/r1543077 I added some generic classes and interfaces
for event filtering and generation. These should be generic enough to implement
standard JCR observation, observation PLUS as described in this issue, more a
more tight integration into Sling like we are discussion in OAK-1120 and
integration with the [OSGi EventAdmin service |
http://markmail.org/message/5g5kl6as4c2hqfgf].
* At http://svn.apache.org/r1543079 I added some filter implementations
specific to JCR observation
* At http://svn.apache.org/r1543080 I implemented JCR observation using the new
classes and interfaces from above.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.13
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13826333#comment-13826333
]
Michael Dürig commented on OAK-1133:
Adding dependency to OAK-190 as we could leverage
{{javax.jcr.observation.EventFilter}} from JCR 2.1 to inject custom filters.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Assignee: Michael Dürig
Labels: observation, performance
Fix For: 0.13
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120, and
[latest discussion on the list|http://markmail.org/message/x2l6tv4m7bxjzqqq].
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813767#comment-13813767
]
Michael Dürig commented on OAK-1133:
I think once the suggested refactorings from OAK-1143 are done, it should be
quite easy to extend the current observation mechanism to cater of most/all of
the requirements listed here and also include things from the patch on OAK-1120
as well as including the {{ExcludeExternal}} semantics from OAK-1121
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813804#comment-13813804
]
Thomas Mueller commented on OAK-1133:
-
listener registrations would include whether they care about local or
external events
very dedicated extra step in the registration API to do so
Yes, that makes sense. It should be relatively hard to register listeners for
external events.
This is not convenience, this is reducing unnecessary work.
It's probably both convenience and reducing work. Convenience is good, if it
results in less and/or clearer code :-)
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810093#comment-13810093
]
Bertrand Delacretaz commented on OAK-1133:
--
From the discussions on list it looks like aggregating events from all the
nodes in a cluster makes scalability difficult, IIUC.
If that's correct, having the option to use an external distribution mechanism
(JMS message queues come to mind) to aggregate cluster events is acceptable
IMO, for cases that require high scalability. Oak could provide a usable
aggregation out of the box, and allow users to move to a more scalable
mechanism where needed.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810344#comment-13810344
]
Thomas Mueller commented on OAK-1133:
-
As far as I understand you want to add more convenience (glob filtering and so
on), but I don't think this will address scalability (the ability to scale the
system to many cluster nodes that concurrently write).
To improve scalability, what is needed is to ensure each event is only
processed in one cluster node, preferably in the cluster node where the change
originated.
To address the scalability problem, see OAK-1121.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810698#comment-13810698
]
Alexander Klimetschek commented on OAK-1133:
Clustered/external events is a somewhat separate topic. I totally agree that
you want to avoid them. AFAICS most events can be handled locally - from my
experience most application use cases require local handling anyway, since
there are other things local to the instance that you depend on, mostly a
sticky session from the web server to ensure users see the data as quickly as
possible.
But still I think an enormous waste is going on if you look at actual listeners
in applications that registers broadly for all events and you need all the
eventing/threading going on, just to figure out in 80% of the cases that this
event can be discarded by the listener. And the listener has to read the
repository data again in a separate session just to do the check. This is not
convenience, this is reducing unnecessary work.
Now the same principle can be used for cluster events, if you need them: I
don't think an external mechanism such as JMS would really help, as you would
add an extra data stream between instances that would need to send *all
events*, since you cannot know if a listener on a target instance is interested
in it or not (you cannot assume the listener code is shared and the specific
registration happens on all cluster nodes, allowing you to filter out events to
just the ones for which there are listeners on the other nodes).
There already is a cluster sync happening between instances and once it arrives
on the target instance, the same approach as proposed here would happen: those
registered filters would run (not sure if the oak {{Observer}} as hook works
here) and only send out events if needed. Of course those registrations/filters
would include whether they care about local or external events. Again different
from today where you have to look at the event as it arrives in the listener to
decide oh this was external, no I don't care after having already wasted
precious resources for that event. And by default a listener would not register
for external events, it would need to be a very dedicated extra step in the
registration API to do so, to discourage accidentally registering for them,
based on the experience that maybe 98% of observation use cases don't need
external events.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar
[jira] [Commented] (OAK-1133) Observation listener PLUS
[
https://issues.apache.org/jira/browse/OAK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810855#comment-13810855
]
Alexander Klimetschek commented on OAK-1133:
Maybe I can find some time to come up with a little prototype, but no
guarantees.
Observation listener PLUS
-
Key: OAK-1133
URL: https://issues.apache.org/jira/browse/OAK-1133
Project: Jackrabbit Oak
Issue Type: New Feature
Components: commons, jcr
Reporter: Alexander Klimetschek
Labels: performance
Oak should provide an *extended and efficient JCR observation listener*
mechanism to support common use cases not handled well by the restricted
options of the JCR observation (only base path, node types and raw events).
Those cases require listeners to register much more broadly and then filter
out their specific cases themselves, thus putting too many events into the
observation system and creating a huge overhead due to asynchronous access to
the modified JCR data to do the filtering. This easily is a big performance
bottleneck with many writes and thus many events.
Previous discussions [on the
list|http://markmail.org/message/oyq7fnfrveceemoh] and in OAK-1120.
The goals should be:
* performance: handle filtering as early as possible, during the commit,
where access to the modified data is already present
* provide robust implementation for typical filtering cases
* provide an asynchronous listener mechanism as in JCR
* minimize effect on the lower levels on Oak (a visible addition in
oak-commons or oak-jcr should be enough)
* for delete events, allow filtering on the to-be-deleted data (currently not
possible in jcr listeners that run after the fact)
* ignore external cluster events by default; have an extra option if you
really want to register for external events
* if possible: design as an extension of the jcr observation to simplify
migration for existing code
* if possible: provide an intelligent listener that can work with pure JCR
(aka Jackrabbit 2) as well, by falling back to in-listener-filtering
* maybe: synchronous option using the same simple interface (instead of raw
Oak plugins itself); however, not sure if there is a benefit if they can only
read data and not change or block the session commit
Typical filtering cases:
- paths with globbing support (for example /content/foo/*/something)
- check for property values (equal, not equal, contains etc.), most
importantly
sling:resourceType in Sling apps
- allow to check properties on child nodes as well, typically jcr:content
- check for any parent/ancestor as well (e.g. change deep inside a node type
= foo structure should be triggered, even if the node with the type wasn't
modified; very important to support efficiently)
- node types (already in jcr observation)
- created/modified/deleted events, separate from move/copy
- and more... a custom filter should be possible to pass through (with
similar access as the {{Observer}})
--
This message was sent by Atlassian JIRA
(v6.1#6144)
