Hi Johannes, I totally agree. This would also mean that we have some kind of admin view, where a user can browse all the error logs, right?
Philipp Am 03.11.21, 20:57 schrieb "Johannes Tex" <[email protected]>: Hi Philipp, we should minimally store the protocol message itself, the status, the date and the source. In the data explorer we should be able to sort by all these values. I think that for production environments it is necessary to monitor the whole installation. Regards Johannes On 2021/10/26 14:17:09 Philipp Zehnder wrote: > Hi Johannes, > > I like the idea of predefined pipelines that are directly started when installing the system. I recently re-activated the pipeline templates feature to directly store data when creating an adapter. We can use this mechanism for that as well. > I think it would also be great to store the log data, my suggestion would be to use the data explorer instead of the live dashboard. > How would you store the data, or what exactly do we need to log and how do we need to represent it? Do you have any thoughts on that? > > I'm not quite sure yet if we should log the entire installation or just the individual elements. > > Philipp > > > On 23. Oct 2021, at 13:34, Johannes Tex <[email protected]> wrote: > > > > Hi Philipp, > > > > I really like your idea in [1] - it's the StreamPipes way :) Everything we can do with pipelines, weshould do with pipelines! Is it possible to predefine pipelines that are already running after installation? IMHO a predefined dashboard would also be necessary. > > > > But... > > Do you want to monitor only the elements this way or the whole installation? If you want to monitor all services this way, the disadvantage is that the UI, pipelines and backend have to be running. If you use the ELK stack (or an other monitor stack), you can still brows the logs even if the UI, backend or/and pe's are not running. > > > > Johannes > > > > On 2021/10/23 08:05:26, Philipp Zehnder <[email protected]> wrote: > >> Hi Johannes, > >> > >> what do you think should we rather use the ELK stack to collect the logs or use the logging mechanism in [1] and transmit the logs over our message broker that we use internally. > >> The advantage would be that the message broker is already there, so we need no additional services. > >> > >> @all Or are there any other ideas how we can collect the logs / errors of the individual services including the information of which StreamPipes element (adapter / processor / …) was responsible. > >> > >> Philipp > >> > >>> On 21. Oct 2021, at 16:50, Johannes Tex <[email protected]> wrote: > >>> > >>> Hi all, > >>> > >>> some time ago I implemented the EventStatisticLogger [1], which is currently not used. The PE wrappers call this logger for each incoming event. The logs were delivered to elasticsearch/kibana via filebeat. > >>> > >>> Regards > >>> Johannes > >>> > >>> [1] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java > >>> > >>> > >>> > >>> On 2021/10/20 17:42:04, Dominik Riemer <[email protected]> wrote: > >>>> Hi Philipp, > >>>> > >>>> I think we can either catch exceptions somewhere in the upper parts of our processing chain and send them to a broker topic, where we collect exceptions similar to the monitoring feature. > >>>> We also had the concept of a StreamPipes-specific logging system, which we can also bring to work so that user-defined logging is available to the core and user UI. I'm not fully aware of the current status of this logger. > >>>> > >>>> Dominik > >>>> > >>>> On 2021/10/20 09:48:01, Philipp Zehnder <[email protected]> wrote: > >>>>> Hi Domink, > >>>>> > >>>>> yes, I think that would also be a good option to transmit the events. > >>>>> I was wondering, how can we intercept the exceptions that occur within a service? > >>>>> > >>>>> Philipp > >>>>> > >>>>>> On 19. Oct 2021, at 22:32, Dominik Riemer <[email protected]> wrote: > >>>>>> > >>>>>> Hi Philipp, > >>>>>> > >>>>>> looks good! I wonder if we should also think about monitoring errors in the adapter and pipeline elements and how to submit the stack trace of exceptions (e.g., an adapter or processing element which throws a runtime error). > >>>>>> > >>>>>> Does it make sense to include an array containing exceptions into the message payload? > >>>>>> > >>>>>> Dominik > >>>>>> > >>>>>> On 2021/10/19 14:24:11, Philipp Zehnder <[email protected]> wrote: > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I’d like to discuss a feature to monitore our pipeline elements, like adapters, processing elements and sinks. > >>>>>>> I created a wiki page with a description of the idea [1]. > >>>>>>> The main idea is to detect if an element is failing (e.g. an adapter stops sending events) and notify the admin of the system. > >>>>>>> > >>>>>>> I’ll create a prototype according to the description in the wiki. If you have any ideas or feature request please write them so that we can discuss them. > >>>>>>> I am looking forward to your feedback on this feature. > >>>>>>> > >>>>>>> Philipp > >>>>>>> > >>>>>>> > >>>>>>> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >> > >> > >
