Hi everyone,
Me and my colleagues are working on a series of pipeline elements that need to
communicate with our external service (a service which is not related to the
StreamPipes).
Basically, we imagined that each of these elements receives an Event, sends
Event data to the external service, this service processes the data, sends it
back to the pipeline element and, finally, pipeline element forwards this data
to the next element in the pipeline. Ideally, the external service processing
and data exchange, both between element & service and between elements, should
be executed before new Event arrives. But we cannot be sure that this is always
going to be the case.
Based on all of this, we singled out several ways of achieving mentioned
communication:
1. The simplest way to achieve this would be to send an HTTP request to the
external service and to wait for its response with the resulting data. This
method raises following questions:
* If pipeline element waits for the response, that means that the main
thread is blocked. If, for whatever reason, this element does not receive the
response before new Event arrives, what would happen to this new Event?
* If element is not able to process Event as soon as it gets sent by the
previous element, does this Event gets lost, or is there a retention period
within used message broker for exchanging Events?
* If there is a retention period, how long is it? Is there an option to
configure it?
* What message broker is used by default?
* We could use a separate thread which would send a request to the
external service, wait for the response and forward it to the next pipeline
element. In this case, the main thread does not get blocked and can receive new
Events if necessary. Would this be a good solution?
2. Implement an active endpoint within each pipeline element to which the
external service sends the results:
* What would be the best possible way to define an endpoint?
* For example, if we use Spring, I suppose that all of the
initialization would go into the onInvocation method of "runtime" class. Would
this block the main thread and disable it from calling onEvent method (because
this thread needs to listen for new requests)? If so, what would the workaround?
* What framework would you recommend?
3. Implement separate component that will be started every time StreamPipes
is started (StreamPipes service). This component would receive responses from
the external service for all the pipeline elements and forward it to the
appropriate one (publish/subscribe architecture between pipeline elements and
said component).
* This way, external service always uses single endpoint to send back
the results to the component.
* Pipeline elements would subscribe to appropriate topic of some message
broker to which the component publishes the results from the external service.
* Would we be able to use one of the existing message broker services
started by the StreamPipes? If yes, what would be the best way of doing this?
According to your experience, what would the best option be? Is there some
other option that we possibly missed?
Any help is greatly appreciated.
Kind regards,
Branislav
