Hi Philipp, I really like your idea in [1] - it's the StreamPipes way :) Everything we can do with pipelines, weshould do with pipelines! Is it possible to predefine pipelines that are already running after installation? IMHO a predefined dashboard would also be necessary.
But... Do you want to monitor only the elements this way or the whole installation? If you want to monitor all services this way, the disadvantage is that the UI, pipelines and backend have to be running. If you use the ELK stack (or an other monitor stack), you can still brows the logs even if the UI, backend or/and pe's are not running. Johannes On 2021/10/23 08:05:26, Philipp Zehnder <[email protected]> wrote: > Hi Johannes, > > what do you think should we rather use the ELK stack to collect the logs or > use the logging mechanism in [1] and transmit the logs over our message > broker that we use internally. > The advantage would be that the message broker is already there, so we need > no additional services. > > @all Or are there any other ideas how we can collect the logs / errors of the > individual services including the information of which StreamPipes element > (adapter / processor / …) was responsible. > > Philipp > > > On 21. Oct 2021, at 16:50, Johannes Tex <[email protected]> wrote: > > > > Hi all, > > > > some time ago I implemented the EventStatisticLogger [1], which is > > currently not used. The PE wrappers call this logger for each incoming > > event. The logs were delivered to elasticsearch/kibana via filebeat. > > > > Regards > > Johannes > > > > [1] > > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java > > > > > > > > On 2021/10/20 17:42:04, Dominik Riemer <[email protected]> wrote: > >> Hi Philipp, > >> > >> I think we can either catch exceptions somewhere in the upper parts of our > >> processing chain and send them to a broker topic, where we collect > >> exceptions similar to the monitoring feature. > >> We also had the concept of a StreamPipes-specific logging system, which we > >> can also bring to work so that user-defined logging is available to the > >> core and user UI. I'm not fully aware of the current status of this logger. > >> > >> Dominik > >> > >> On 2021/10/20 09:48:01, Philipp Zehnder <[email protected]> wrote: > >>> Hi Domink, > >>> > >>> yes, I think that would also be a good option to transmit the events. > >>> I was wondering, how can we intercept the exceptions that occur within a > >>> service? > >>> > >>> Philipp > >>> > >>>> On 19. Oct 2021, at 22:32, Dominik Riemer <[email protected]> wrote: > >>>> > >>>> Hi Philipp, > >>>> > >>>> looks good! I wonder if we should also think about monitoring errors in > >>>> the adapter and pipeline elements and how to submit the stack trace of > >>>> exceptions (e.g., an adapter or processing element which throws a > >>>> runtime error). > >>>> > >>>> Does it make sense to include an array containing exceptions into the > >>>> message payload? > >>>> > >>>> Dominik > >>>> > >>>> On 2021/10/19 14:24:11, Philipp Zehnder <[email protected]> wrote: > >>>>> Hi all, > >>>>> > >>>>> I’d like to discuss a feature to monitore our pipeline elements, like > >>>>> adapters, processing elements and sinks. > >>>>> I created a wiki page with a description of the idea [1]. > >>>>> The main idea is to detect if an element is failing (e.g. an adapter > >>>>> stops sending events) and notify the admin of the system. > >>>>> > >>>>> I’ll create a prototype according to the description in the wiki. If > >>>>> you have any ideas or feature request please write them so that we can > >>>>> discuss them. > >>>>> I am looking forward to your feedback on this feature. > >>>>> > >>>>> Philipp > >>>>> > >>>>> > >>>>> [1] > >>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025 > >>>>> > >>>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025> > >>>>> > >>>>> > >>> > >>> > >> > >
