Hi Johannes, what do you think should we rather use the ELK stack to collect the logs or use the logging mechanism in [1] and transmit the logs over our message broker that we use internally. The advantage would be that the message broker is already there, so we need no additional services.
@all Or are there any other ideas how we can collect the logs / errors of the individual services including the information of which StreamPipes element (adapter / processor / …) was responsible. Philipp > On 21. Oct 2021, at 16:50, Johannes Tex <[email protected]> wrote: > > Hi all, > > some time ago I implemented the EventStatisticLogger [1], which is currently > not used. The PE wrappers call this logger for each incoming event. The logs > were delivered to elasticsearch/kibana via filebeat. > > Regards > Johannes > > [1] > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java > > > > On 2021/10/20 17:42:04, Dominik Riemer <[email protected]> wrote: >> Hi Philipp, >> >> I think we can either catch exceptions somewhere in the upper parts of our >> processing chain and send them to a broker topic, where we collect >> exceptions similar to the monitoring feature. >> We also had the concept of a StreamPipes-specific logging system, which we >> can also bring to work so that user-defined logging is available to the core >> and user UI. I'm not fully aware of the current status of this logger. >> >> Dominik >> >> On 2021/10/20 09:48:01, Philipp Zehnder <[email protected]> wrote: >>> Hi Domink, >>> >>> yes, I think that would also be a good option to transmit the events. >>> I was wondering, how can we intercept the exceptions that occur within a >>> service? >>> >>> Philipp >>> >>>> On 19. Oct 2021, at 22:32, Dominik Riemer <[email protected]> wrote: >>>> >>>> Hi Philipp, >>>> >>>> looks good! I wonder if we should also think about monitoring errors in >>>> the adapter and pipeline elements and how to submit the stack trace of >>>> exceptions (e.g., an adapter or processing element which throws a runtime >>>> error). >>>> >>>> Does it make sense to include an array containing exceptions into the >>>> message payload? >>>> >>>> Dominik >>>> >>>> On 2021/10/19 14:24:11, Philipp Zehnder <[email protected]> wrote: >>>>> Hi all, >>>>> >>>>> I’d like to discuss a feature to monitore our pipeline elements, like >>>>> adapters, processing elements and sinks. >>>>> I created a wiki page with a description of the idea [1]. >>>>> The main idea is to detect if an element is failing (e.g. an adapter >>>>> stops sending events) and notify the admin of the system. >>>>> >>>>> I’ll create a prototype according to the description in the wiki. If you >>>>> have any ideas or feature request please write them so that we can >>>>> discuss them. >>>>> I am looking forward to your feedback on this feature. >>>>> >>>>> Philipp >>>>> >>>>> >>>>> [1] >>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025 >>>>> >>>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025> >>>>> >>>>> >>> >>> >>
