Hi, I've worked on a new feature to collect monitoring infos for pipelines and adapters. In the current release, only very basic monitoring info was displayed for running pipelines, which only worked when using Kafka as message broker and often displayed strange values as we were receiving the info directly from the Kafka admin client. The new implementation, independent from the underlying broker, collects metrics for adapters and pipeline elements (messages received, messages sent and last timestamp of received messages) and makes runtime exceptions available in the UI. Exceptions are shown in the "Logs" section within the details view of pipelines and adapters. On the SDK side, there is also a new method in the RuntimeContext of processors and sinks to forward custom exceptions to the core.
I hope this greatly improves transparency for users as runtime exceptions were previously only available in the log files. There are two things left: * I know that some community members have asked for a central REST endpoint which provides monitoring info in a single document, e.g., to read this from external monitoring software. This shouldn't be hard to implement now and I'm happy to assist. * Exceptions for adapters are currently very basic, as there is now easy way to globally catch exceptions from adapters. I'll try to improve the StreamPipes Connect API and codebase first, which should then also ease the definition of integration tests Zike has started to work on. I'll merge the changes into dev soon and I'm happy for any feedback! Cheers Dominik
