GitHub user dominikriemer edited a discussion: Integrate service discovery into core?
Hi, currently, StreamPipes has a dependency on Consul for two reasons: * Extensions services register at Consul and the core makes use of Consul's service discovery to find endpoints that support pipeline elements and adapters * The core uses Consul's key/value store to store configs such as email, authentication settings, broker settings and more. In addition, extensions can define configurations which are also stored in Consul. I think it should be pretty easy to integrate the service discovery logic directly into the core and store the configurations in CouchDB, which would make our Consul dependency obsolete, reducing system complexity. ## Approach The approach would be: * Extensions services register at the core upon startup, using a new endpoint which stores the service configuration in CouchDB. The core's hostname would be `backend` by default in the docker-compose installation, link to the backend service in k8s and could be overridden with an environment variable similar to `SP_CONSUL_LOCATION` for development purposes. * The core then uses its own service discovery mechanism based on stored service configurations when pipelines and adapters are started. * All configurations of the core which are currently stored in Consul would move to a new CouchDB database `core-configuration`. * Extensions services could store their configs in another CouchDB database which can be accessed over the client from the core. ## Migration * We don't need a migration for the service discovery change, assuming that extensions services and the core are running the same StreamPipes version * For the key/value store configurations, we need a migration. We could add a migration in release 0.93.0 which transfers all Consul configs to the CouchDB database, and remove Consul in the release thereafter. What do ou think of this idea? GitHub link: https://github.com/apache/streampipes/discussions/1811 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
