GitHub user spetz added a comment to the discussion: Connectors to build a modular, plugin system
After some initial research, discussions and the quick prototyping, we came up with the following conclusions: - the plugin needs to be able to deal with the potential back pressure (e.g. lots of received data that needs to be transformed and then stored in some external DB/Service) - instead of pushing the raw message data to the plugin (even with IPC), the Iggy server would simply notify about the updated offset in the particular stream->topic->partition (depending on the subscribed topics) - there will be other types of notifications as well, such as deleted segments, created users, etc. - connector will pull the data as if it was the regular stream consumer (once again, pull model instead of the push one) - to read the data as efficiently as possible without the additional overhead, the connector will keep the opened file descriptor in read-only mode to the subscribed resources (the segment is the actual file storing an append-only log of the messages) - there will be no interference with the server having another FD in append-mode, as the plugin will only read the committed records - thanks to the zero-copy deserialization, and the upcoming support for DirectIO and io_uring, there will be no issues with page cache, and the reads will be extremely fast - we will start with the local connectors only for now, written purely in Rust, as the separate applications - having the separate processes, instead of UDF directly embedded/invoked by the server, will make it easier to deal with the potential errors (invalid plugin won't crash the streaming server) As always, looking forward to your feedback :) GitHub link: https://github.com/apache/iggy/discussions/1670#discussioncomment-12877795 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
