GitHub user spetz added a comment to the discussion: Configuration Storage Trait for Connectors Runtime
Hey, This is a good idea, since currently, the connectors store the configuration in the local files, making it unusable with distributed computing (scaling out the instances). IMHO, to begin with, we would need a simple trait allowing us to upsert (save), get, delete, and list versions (since each connector has an independent config, the easy way to detect the changes would be by storing and comparing config versions). And actually, the get and delete functions could take an optional version parameter (if not provided, fallback to the current one) so that it'd be easy to store/rollback to previous configurations if needed. Additional extensions like audit logs, etc., could be implemented in the future. The connectors API (which are now read-only endpoints for sink and source plugins) should use the particular config storage trait to perform all the actions, and on top of this, there could be a basic background job with a configurable interval, polling for changes (e.g. by comparing the mentioned config versions) to automatically update the configs stored in memory. GitHub link: https://github.com/apache/iggy/discussions/2343#discussioncomment-14955792 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
