[CONF] Apache Sling > Higher level observation services

[confluence]

Higher level observation services Page edited by Bertrand Delacretaz Changes (68) {style} .deflist h4 { margin-top: 0; font-size:100%; font-weight:normal; font-style:italic; color:#888888; } .deflist h4 + p { margin-top: 0; } .deflist p { margin-left: 2em; } ... {div:class=deflist} h4. Scenario In-memory data structures, or "compiled" versions of some content, are created when content changes. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. h4. Frequency Content changes are usually not very frequent, for the above typical uses. ... {div} h2. Content Export, Replication to Remote Systems {div:class=deflist} h4. Scenario Content is exported as a file or pushed to a remote system when it changes. {div} This is similar to the Cached Content pattern. h2. Content Ingestion {div:class=deflist} h4. Scenario Files that are dropped into the repository are parsed or processed, resulting in content changes and/or workflow events. h4. Typical uses Ingesting digital assets, parsing incoming email or other structured files. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. A file appears in a watched folder. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Process the file, create the corresponding content, move the file to a "processed" or "rejected" folder. h4. Frequency Content changes are usually not very frequent, for the above typical uses. Depends on the application. h4. Performance Some latency between content changes and processing is usually not a problem. Some latency is usually not a problem, but some applications need to process large number of files quickly. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Processing partially saved files too early can be a problem, depending on how files are added to the repository. {div} h2. Content Export h2. Content Tree Replication {div:class=deflist} h4. Scenario Two or more content trees are kept in sync, usually with customizable mappings and transformations. h4. Typical uses Management of federations of websites, which have some common parts and some specific parts. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Replicate the source tree to the target tree(s), optionally applying customizable content transformations. h4. Frequency Content changes are usually not very frequent, for the above typical uses. Source events might be quite frequent depending on authoring activity. h4. Performance Some latency between content changes and processing is usually not a problem. Tree transformations might be costly and usually need to run as background jobs. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. An explosion in the number of application-level and repository-level operations is possible depending on the shape of the content tree federation and on the frequency of source content changes. {div} h2. Aggregation of changes {div:class=deflist} h4. Scenario Collect a number of change events over time and/or for a content subtree, and provide an aggregated view. h4. Typical uses Detect and act on changes to digital assets, without reacting to each and every small change. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Store and aggregate events and deliver the results on demand. h4. Frequency Content changes are usually not very frequent, for the above typical uses. Might be quite frequent on a busy content tree. h4. Performance Some latency between content changes and processing is usually not a problem. Aggregating events efficiently, as well as storing them until they're not needed anymore, can impact performance. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Might create a noticeable load on the eventing/observation system, if listening to many detailed events. {div} h2. Consistency Checks and Fixes {div:class=deflist} h4. Scenario Watch specific content subtrees for specific changes (nodes moved etc.) and react to them to avoid inconsistencies in the content. h4. Typical uses Adapt paths that point to other pieces of content when content moves around. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Modify content to keep it consistent h4. Frequency Content changes are usually not very frequent, for the above typical uses. Usually not very frequent as that's mostly meant to handle edge cases. h4. Performance Some latency between content changes and processing is usually not a problem. A time window during which content can be seen as inconsistent is often unavoidable, keeping that window small is useful. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Content must not be modified by JCR listeners, that should happen asynchronously if using JCR observation. {div} h2. Content Replication h2. Workflow/Job Trigger {div:class=deflist} h4. Scenario Trigger workflows and jobs when content changes. h4. Typical uses The unix print queue system is a good example, with folders named _incoming_, _printing_, _done_, _rejected_ that store print job definitions. h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. Detection of new content items in the _incoming_ folder(s). h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Execute the corresponding tasks and move the message nodes according to the results. h4. Frequency Content changes are usually not very frequent, for the above typical uses. Depends on the application. h4. Performance Some latency between content changes and processing is usually not a problem. Depends on the application. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Locking must be used if several job processors are competing for _incoming_ jobs. Distributed processing of those jobs introduces cluster management requirements. {div} h2. Message Queue {div:class=deflist} h4. Trigger Scenario Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. h4. Frequency Content changes are usually not very frequent, for the above typical uses. h4. Performance Some latency between content changes and processing is usually not a problem. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Implement a simple message queue backed by a content repository subtree. {div} h2. Workflow/Job Trigger {div:class=deflist} h4. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. h4. Action Clear an internal cache that is rebuilt the next time someone needs it. h4. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. h4. Frequency Content changes are usually not very frequent, for the above typical uses. h4. Performance Some latency between content changes and processing is usually not a problem. h4. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. {div} This is very similar to the Workflow/Job trigger use case: messages are exchanged between producers and consumers. Full Content Analyzing how we use observation in our Sling-based apps shows a number of recurring patterns, described in this page. Using JCR observation directly or Sling OSGi events does not make a big difference in the final results, but the implementations are very different. The commit hooks provided by http://jackrabbit.apache.org/oak/ provide yet another way of observing content changes, which might be more efficient or scalable in some cases. Being able to express these common observation patterns as higher-level services, if we can do that, would allow for switching the underlying implementation seamlessly, and would also help promote best practices in how we use events, by minimizing the amount of code to write at the application level. Observation usage patterns Let's list the frequent patterns that we see w.r.t observing changes in content. Cached Content Scenario In-memory data structures, or "compiled" versions of some content, are created when content changes. Typical uses Configurations, CSS/_javascript_ processing, Sling installer, etc. Trigger Changes in content under specific paths (or path regexps), with optional restrictions on node types and/or event types. Action Clear an internal cache that is rebuilt the next time someone needs it. Frequency Content changes are usually not very frequent, for the above typical uses. Performance Some latency between content changes and processing is usually not a problem. Potential issues N requests coming just after clearing the cache should cause just one cache rebuild, not N. Content Export, Replication to Remote Systems Scenario Content is exported as a file or pushed to a remote system when it changes. This is similar to the Cached Content pattern. Content Ingestion Scenario Files that are dropped into the repository are parsed or processed, resulting in content changes and/or workflow events. Typical uses Ingesting digital assets, parsing incoming email or other structured files. Trigger A file appears in a watched folder. Action Process the file, create the corresponding content, move the file to a "processed" or "rejected" folder. Frequency Depends on the application. Performance Some latency is usually not a problem, but some applications need to process large number of files quickly. Potential issues Processing partially saved files too early can be a problem, depending on how files are added to the repository. Content Tree Replication Scenario Two or more content trees are kept in sync, usually with customizable mappings and transformations. Typical uses Management of federations of websites, which have some common parts and some specific parts. Trigger Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. Action Replicate the source tree to the target tree(s), optionally applying customizable content transformations. Frequency Source events might be quite frequent depending on authoring activity. Performance Tree transformations might be costly and usually need to run as background jobs. Potential issues An explosion in the number of application-level and repository-level operations is possible depending on the shape of the content tree federation and on the frequency of source content changes. Aggregation of changes Scenario Collect a number of change events over time and/or for a content subtree, and provide an aggregated view. Typical uses Detect and act on changes to digital assets, without reacting to each and every small change. Trigger Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. Action Store and aggregate events and deliver the results on demand. Frequency Might be quite frequent on a busy content tree. Performance Aggregating events efficiently, as well as storing them until they're not needed anymore, can impact performance. Potential issues Might create a noticeable load on the eventing/observation system, if listening to many detailed events. Consistency Checks and Fixes Scenario Watch specific content subtrees for specific changes (nodes moved etc.) and react to them to avoid inconsistencies in the content. Typical uses Adapt paths that point to other pieces of content when content moves around. Trigger Fine-grained detection of changes in a tree of content, based on paths, path regexps, node types or any other meaningful property. Action Modify content to keep it consistent Frequency Usually not very frequent as that's mostly meant to handle edge cases. Performance A time window during which content can be seen as inconsistent is often unavoidable, keeping that window small is useful. Potential issues Content must not be modified by JCR listeners, that should happen asynchronously if using JCR observation. Workflow/Job Trigger Scenario Trigger workflows and jobs when content changes. Typical uses The unix print queue system is a good example, with folders named incoming, printing, done, rejected that store print job definitions. Trigger Detection of new content items in the incoming folder(s). Action Execute the corresponding tasks and move the message nodes according to the results. Frequency Depends on the application. Performance Depends on the application. Potential issues Locking must be used if several job processors are competing for incoming jobs. Distributed processing of those jobs introduces cluster management requirements. Message Queue Scenario Implement a simple message queue backed by a content repository subtree. This is very similar to the Workflow/Job trigger use case: messages are exchanged between producers and consumers. Change Notification Preferences View Online | View Changes | Add Comment