Hi Philipp, we should minimally store the protocol message itself, the status, the date and the source. In the data explorer we should be able to sort by all these values.
I think that for production environments it is necessary to monitor the whole installation. Regards Johannes On 2021/10/26 14:17:09 Philipp Zehnder wrote: > Hi Johannes, > > I like the idea of predefined pipelines that are directly started when > installing the system. I recently re-activated the pipeline templates feature > to directly store data when creating an adapter. We can use this mechanism > for that as well. > I think it would also be great to store the log data, my suggestion would be > to use the data explorer instead of the live dashboard. > How would you store the data, or what exactly do we need to log and how do we > need to represent it? Do you have any thoughts on that? > > I'm not quite sure yet if we should log the entire installation or just the > individual elements. > > Philipp > > > On 23. Oct 2021, at 13:34, Johannes Tex <[email protected]> wrote: > > > > Hi Philipp, > > > > I really like your idea in [1] - it's the StreamPipes way :) Everything we > > can do with pipelines, weshould do with pipelines! Is it possible to > > predefine pipelines that are already running after installation? IMHO a > > predefined dashboard would also be necessary. > > > > But... > > Do you want to monitor only the elements this way or the whole > > installation? If you want to monitor all services this way, the > > disadvantage is that the UI, pipelines and backend have to be running. If > > you use the ELK stack (or an other monitor stack), you can still brows the > > logs even if the UI, backend or/and pe's are not running. > > > > Johannes > > > > On 2021/10/23 08:05:26, Philipp Zehnder <[email protected]> wrote: > >> Hi Johannes, > >> > >> what do you think should we rather use the ELK stack to collect the logs > >> or use the logging mechanism in [1] and transmit the logs over our message > >> broker that we use internally. > >> The advantage would be that the message broker is already there, so we > >> need no additional services. > >> > >> @all Or are there any other ideas how we can collect the logs / errors of > >> the individual services including the information of which StreamPipes > >> element (adapter / processor / …) was responsible. > >> > >> Philipp > >> > >>> On 21. Oct 2021, at 16:50, Johannes Tex <[email protected]> wrote: > >>> > >>> Hi all, > >>> > >>> some time ago I implemented the EventStatisticLogger [1], which is > >>> currently not used. The PE wrappers call this logger for each incoming > >>> event. The logs were delivered to elasticsearch/kibana via filebeat. > >>> > >>> Regards > >>> Johannes > >>> > >>> [1] > >>> https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java > >>> > >>> > >>> > >>> On 2021/10/20 17:42:04, Dominik Riemer <[email protected]> wrote: > >>>> Hi Philipp, > >>>> > >>>> I think we can either catch exceptions somewhere in the upper parts of > >>>> our processing chain and send them to a broker topic, where we collect > >>>> exceptions similar to the monitoring feature. > >>>> We also had the concept of a StreamPipes-specific logging system, which > >>>> we can also bring to work so that user-defined logging is available to > >>>> the core and user UI. I'm not fully aware of the current status of this > >>>> logger. > >>>> > >>>> Dominik > >>>> > >>>> On 2021/10/20 09:48:01, Philipp Zehnder <[email protected]> wrote: > >>>>> Hi Domink, > >>>>> > >>>>> yes, I think that would also be a good option to transmit the events. > >>>>> I was wondering, how can we intercept the exceptions that occur within > >>>>> a service? > >>>>> > >>>>> Philipp > >>>>> > >>>>>> On 19. Oct 2021, at 22:32, Dominik Riemer <[email protected]> wrote: > >>>>>> > >>>>>> Hi Philipp, > >>>>>> > >>>>>> looks good! I wonder if we should also think about monitoring errors > >>>>>> in the adapter and pipeline elements and how to submit the stack trace > >>>>>> of exceptions (e.g., an adapter or processing element which throws a > >>>>>> runtime error). > >>>>>> > >>>>>> Does it make sense to include an array containing exceptions into the > >>>>>> message payload? > >>>>>> > >>>>>> Dominik > >>>>>> > >>>>>> On 2021/10/19 14:24:11, Philipp Zehnder <[email protected]> wrote: > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I’d like to discuss a feature to monitore our pipeline elements, like > >>>>>>> adapters, processing elements and sinks. > >>>>>>> I created a wiki page with a description of the idea [1]. > >>>>>>> The main idea is to detect if an element is failing (e.g. an adapter > >>>>>>> stops sending events) and notify the admin of the system. > >>>>>>> > >>>>>>> I’ll create a prototype according to the description in the wiki. If > >>>>>>> you have any ideas or feature request please write them so that we > >>>>>>> can discuss them. > >>>>>>> I am looking forward to your feedback on this feature. > >>>>>>> > >>>>>>> Philipp > >>>>>>> > >>>>>>> > >>>>>>> [1] > >>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025 > >>>>>>> > >>>>>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >> > >> > >
