bossenti commented on issue #2001: URL: https://github.com/apache/streampipes/issues/2001#issuecomment-1782520174
## Introduce the ability to migrate existing adapters & PE >[!IMPORTANT] > The current implementation only allows modifications that affect the configuration of > an element. Changes that affect the event stream are out of scope for this iteration > (see more at next steps). This can be split into major parts: - update description for new elements: adapter, processor, sinks - update descriptions for existing elements: adapter, processor, sinks ### Rationale As of now, a user cannot simply upgrade to a recent StreamPipes version if this version introduces changes in a pipeline element (adapter, processor, or sink). To create new instances of an affected pipeline element the user needs to update them first via the UI. Secondly, existing instances of an affected pipeline element were required to be recreated manually. Therefore, we now introduce so-called migrations to update both the descriptions used for new elements and the existing elements automatically in the background. ### General Approach A developer is now enabled to define one or multiple `Migration` for a pipeline element. Therefore, one of `AdapterMigrator`, `DataProcessorMigrator`, or `DataSinkMigrator` from `streampipes-extensions-api` needs to be implemented. In addition, the migration needs to be added to the service definition in the extensions model submitter, e.g., `AllExtensionsIIoTInit`. From now on, the migration is fully automatically handled according to the following flow:  ### Update Description for New Elements For every migration that is sent from the extensions service to the core, the core does update the description of the corresponding element in the storage (couchDB). This is done by requesting the current configuration from the extensions service and overriding the existing one in the storage. This automatic update of all element descriptions makes the manual update in the UI redundant. Thus we can remove this functionality as an upcoming step. ### Update Descriptions for Existing Elements For every migration that is sent from the extensions service to the core, the core checks if there are any existing elements that are affected by this migration. If so, the migration is requested by the extensions service which then executes the migration on behalf of the core. In case this is successful, the core can simply update the configuration. If not, we stop the corresponding pipeline element. For processors & sinks, the corresponding pipeline is even marked with "needs attention". However, (at least for adapters) this is not yet the ideal solution for handling migration failures. ### Important Information - The description of all pipeline elements is extended by an additional field `version` which needs to be provided in the future. However, we can make this change step by step since the version always defaults to `0`. - The order in which a migration is registered at the service definition does not affect anything. The order of the migrations is always ensured so that the earliest migrations are always applied first. - Before a migration is applied to an element, it is checked if this element needs the migration. So we don't need to worry about migrations being applied multiple times to the same element. ### Next steps The following steps need to be handled (at least) to fully support migrations at StreamPipes: - Handle failed migrations properly - Handle case for element that is older than all migrations provided - Remove update functionality from UI - Assets need to be updated as well (only for adapters) - Enable migrations for scenarios where the event stream is affected - Handle cases with multiple extension services providing the same pipeline elements -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
