[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[ https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116227#comment-16116227 ] Konrad Windszus commented on SLING-6745: For the record, the modules can be found now `contrib/commons/mom` (https://github.com/apache/sling/tree/trunk/contrib/commons/mom). > kafka-based sling.event.api implementation > -- > > Key: SLING-6745 > URL: https://issues.apache.org/jira/browse/SLING-6745 > Project: Sling > Issue Type: New Feature >Reporter: Stefan Egli >Assignee: Stefan Egli > > In job handling to scale for larger deployment it is essential to be able to > execute a job outside of the local instance. This can be in another instance > in the same cluster (ie when ontop of documentMk) or outside of the cluster > (in AEM eg via > [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]). > In both cases Sling Event (Resource) stores the job in the repository > (/var/eventing/jobs). > It could be useful to have another variant of job execution that is managed > entirely outside of the repository. With SLING-5645 a new API was created to > support messaging-based implementations that would fit in this category as > they map a job to a (one or a few) message(s). > This new SLING-5645 Job-API is not entirely compatible with sling.event.api > though. > This ticket is to track an effort to provide a messaging-based implementation > for sling.event.api - namely for Kafka. The goal is to become a drop-in > replacement of sling event without much need for change on the user side of > the sling.event.api. > This might not right away become part of sling, but might start off in the > contrib section - to underscore that it's not something supported, at least > as of yet. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[ https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949270#comment-15949270 ] Stefan Egli commented on SLING-6745: bq. As a summary : I think it doesn't make sense to reuse the MoM-based Jobs API - but it might make sense to use the MoM API/SPI. I'm taking that back for now. Perhaps it's feasible to build ontop of mom/jobs/core I have to see. I think where my concern is that it could be confusing to have 2 'job apis' in the same deployment and that the two might interfere. But 'building ontop of mom/jobs/core' could also be done by embedding it - thus not really exposing its API. > kafka-based sling.event.api implementation > -- > > Key: SLING-6745 > URL: https://issues.apache.org/jira/browse/SLING-6745 > Project: Sling > Issue Type: New Feature >Reporter: Stefan Egli >Assignee: Stefan Egli > > In job handling to scale for larger deployment it is essential to be able to > execute a job outside of the local instance. This can be in another instance > in the same cluster (ie when ontop of documentMk) or outside of the cluster > (in AEM eg via > [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]). > In both cases Sling Event (Resource) stores the job in the repository > (/var/eventing/jobs). > It could be useful to have another variant of job execution that is managed > entirely outside of the repository. With SLING-5645 a new API was created to > support messaging-based implementations that would fit in this category as > they map a job to a (one or a few) message(s). > This new SLING-5645 Job-API is not entirely compatible with sling.event.api > though. > This ticket is to track an effort to provide a messaging-based implementation > for sling.event.api - namely for Kafka. The goal is to become a drop-in > replacement of sling event without much need for change on the user side of > the sling.event.api. > This might not right away become part of sling, but might start off in the > contrib section - to underscore that it's not something supported, at least > as of yet. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[ https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949040#comment-15949040 ] Stefan Egli commented on SLING-6745: [~ianeboston], the MoM-based Job API has definitely been considered, as a kafka variant based on that is pretty straight-forward. However there's the backwards-compatibility argument that came up: there are many existing users of the Sling Event based Job API which would have to be ported to the Mom-based Job API - and the aim is to avoid requiring user adjuments as well as having 2 different APIs in the same deployment. This is why the suggestion was with SLING-6739 to split sling.event into sling.event.api and sling.event.resource (in a way that users of the API wouldn't notice anything, ie code compatible). This would then open the way for other implementations of sling.event.api - such as this kafka variant. Now with regards to this new kafka-based implementation and the MoM-based API & implementation I see the following different variants: # A bridge implementation could map sling.event.api to [sling.job|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jobs/core] and fill in the gaps. One such gap is as mentioned the support for querying for jobs - which is what you also pointed out. That could be added to this bridge. The implementation of the kafka part would then be based on the sling.mom SPI. # An implementation that uses the [MoM API|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/api] for the messaging part, but doesn't use the [MoM-based Sling Jobs API|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jobs/core]). Additionally it would add the job query part, same as above. # just reuse (ie copy + modify) code from the MoM-based implementation (not the API) and marry that with the job query part. Then implement everything for Kafka. # same as 3. but define a different MoM API. I'm not sure yet which is the best option, but 1) a mix of sling.event.api and sling.job API (the [org.apache.sling.jobs|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jobs/core/src/main/java/org/apache/sling/jobs] package) seems less favorable to me. Whether 2. is possible needs to be seen. As a summary : I think it doesn't make sense to reuse the [MoM-based Jobs API|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jobs/core] - but it might make sense to use the [MoM API/SPI|https://github.com/apache/sling/tree/trunk/contrib/commons/mom/api]. Wdyt? > kafka-based sling.event.api implementation > -- > > Key: SLING-6745 > URL: https://issues.apache.org/jira/browse/SLING-6745 > Project: Sling > Issue Type: New Feature >Reporter: Stefan Egli >Assignee: Stefan Egli > > In job handling to scale for larger deployment it is essential to be able to > execute a job outside of the local instance. This can be in another instance > in the same cluster (ie when ontop of documentMk) or outside of the cluster > (in AEM eg via > [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]). > In both cases Sling Event (Resource) stores the job in the repository > (/var/eventing/jobs). > It could be useful to have another variant of job execution that is managed > entirely outside of the repository. With SLING-5645 a new API was created to > support messaging-based implementations that would fit in this category as > they map a job to a (one or a few) message(s). > This new SLING-5645 Job-API is not entirely compatible with sling.event.api > though. > This ticket is to track an effort to provide a messaging-based implementation > for sling.event.api - namely for Kafka. The goal is to become a drop-in > replacement of sling event without much need for change on the user side of > the sling.event.api. > This might not right away become part of sling, but might start off in the > contrib section - to underscore that it's not something supported, at least > as of yet. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[ https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949003#comment-15949003 ] Oliver Lietz commented on SLING-6745: - bq. Wouldnt it be better to improve the MoM API and fix where it doesn't work ? That sounds reasonable. Would it be possible to provide a bridge or layer to provide compatibility with Sling's existing Event/Job API? > kafka-based sling.event.api implementation > -- > > Key: SLING-6745 > URL: https://issues.apache.org/jira/browse/SLING-6745 > Project: Sling > Issue Type: New Feature >Reporter: Stefan Egli >Assignee: Stefan Egli > > In job handling to scale for larger deployment it is essential to be able to > execute a job outside of the local instance. This can be in another instance > in the same cluster (ie when ontop of documentMk) or outside of the cluster > (in AEM eg via > [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]). > In both cases Sling Event (Resource) stores the job in the repository > (/var/eventing/jobs). > It could be useful to have another variant of job execution that is managed > entirely outside of the repository. With SLING-5645 a new API was created to > support messaging-based implementations that would fit in this category as > they map a job to a (one or a few) message(s). > This new SLING-5645 Job-API is not entirely compatible with sling.event.api > though. > This ticket is to track an effort to provide a messaging-based implementation > for sling.event.api - namely for Kafka. The goal is to become a drop-in > replacement of sling event without much need for change on the user side of > the sling.event.api. > This might not right away become part of sling, but might start off in the > contrib section - to underscore that it's not something supported, at least > as of yet. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[
https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948945#comment-15948945
]
Ian Boston commented on SLING-6745:
---
The aim of SLING-5645 was to create an API that was compatible with
sling.event.api, however it was discovered while doing that work and testing in
a distributed environment that some features of the sling.event.api were
created with a single instance in mind and therefore did not fit a distributed
model. Once of those features was the ability to query event via the
distributed API. Looking deeper into distributed event systems like RabbitMQ,
AMQ etc the ability to randomly query the contents of an event queue is
generally not supported. For that reason SLING-5645 dropped that feature and
focused on what was supported. The API that was created is 2 layered. The outer
layer has no dependencies on any other API standard. The Inner layer requires
the JMS API and uses both Topic and Queue concepts. Most messaging providers
have Java clients that support the JMS API so this was considered to be a safe
choice. For example, AWS SMS has a Java JMS Client library.
For providers that do support JMS, all that is needed is this class
{code}
@ProviderType
public interface ConnectionFactoryService {
/**
*
* Get a connection factory.
* @return the connection factory.
*/
ConnectionFactory getConnectionFactory();
}
{code}
see
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/jms/ConnectionFactoryService.java
for example ActiveMQ
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/amq/ActiveMQConnectionFactoryService.java
If Kafka does not have a JMS Client then the following SPI needs to be
implemented.
https://github.com/apache/sling/tree/trunk/contrib/commons/mom/api/src/main/java/org/apache/sling/mom
Provided Kafka has the concept of Topics and Quues then this is relatively
trivial as in
https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/jms/impl
-
Obviously there is nothing to stop anyone reimplementing an alternative MoM API
in your own way as a competing implementation to address the current Sling API.
Although...
Wouldnt it be better to improve the MoM API and fix where it doesn't work ?
There are placeholders for the query operations.
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jobs/core/src/main/java/org/apache/sling/jobs/JobManager.java#L60
I had thought if these were required an independent service would subscribe to
topics to maintain a list of active jobs. There is a topic for job status.
> kafka-based sling.event.api implementation
> --
>
> Key: SLING-6745
> URL: https://issues.apache.org/jira/browse/SLING-6745
> Project: Sling
> Issue Type: New Feature
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>
> In job handling to scale for larger deployment it is essential to be able to
> execute a job outside of the local instance. This can be in another instance
> in the same cluster (ie when ontop of documentMk) or outside of the cluster
> (in AEM eg via
> [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]).
> In both cases Sling Event (Resource) stores the job in the repository
> (/var/eventing/jobs).
> It could be useful to have another variant of job execution that is managed
> entirely outside of the repository. With SLING-5645 a new API was created to
> support messaging-based implementations that would fit in this category as
> they map a job to a (one or a few) message(s).
> This new SLING-5645 Job-API is not entirely compatible with sling.event.api
> though.
> This ticket is to track an effort to provide a messaging-based implementation
> for sling.event.api - namely for Kafka. The goal is to become a drop-in
> replacement of sling event without much need for change on the user side of
> the sling.event.api.
> This might not right away become part of sling, but might start off in the
> contrib section - to underscore that it's not something supported, at least
> as of yet.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[ https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948787#comment-15948787 ] Stefan Egli commented on SLING-6745: Perhaps part of SLING-5645 can be reused for this approach. What would be additional is keeping/providing support for 'job queries' - and that could be supported via a new Job Query Service SPI. > kafka-based sling.event.api implementation > -- > > Key: SLING-6745 > URL: https://issues.apache.org/jira/browse/SLING-6745 > Project: Sling > Issue Type: New Feature >Reporter: Stefan Egli >Assignee: Stefan Egli > > In job handling to scale for larger deployment it is essential to be able to > execute a job outside of the local instance. This can be in another instance > in the same cluster (ie when ontop of documentMk) or outside of the cluster > (in AEM eg via > [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]). > In both cases Sling Event (Resource) stores the job in the repository > (/var/eventing/jobs). > It could be useful to have another variant of job execution that is managed > entirely outside of the repository. With SLING-5645 a new API was created to > support messaging-based implementations that would fit in this category as > they map a job to a (one or a few) message(s). > This new SLING-5645 Job-API is not entirely compatible with sling.event.api > though. > This ticket is to track an effort to provide a messaging-based implementation > for sling.event.api - namely for Kafka. The goal is to become a drop-in > replacement of sling event without much need for change on the user side of > the sling.event.api. > This might not right away become part of sling, but might start off in the > contrib section - to underscore that it's not something supported, at least > as of yet. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
