This is an automated email from the ASF dual-hosted git repository. bossenti pushed a commit to branch provide-docs-for-migration in repository https://gitbox.apache.org/repos/asf/streampipes-website.git
commit 879cb84c62d98ed84a114143294516b21be758f9 Author: bossenti <[email protected]> AuthorDate: Mon Oct 30 08:27:25 2023 +0100 docs: add documentation section for migrations --- docs/06_extend-sdk-migrations.md | 179 +++++++++++++++++++++ website-v2/sidebars.json | 1 + website-v2/src/css/customTheme.css | 7 + .../img/06_sdk_migrations/migration-flow.png | Bin 0 -> 630350 bytes 4 files changed, 187 insertions(+) diff --git a/docs/06_extend-sdk-migrations.md b/docs/06_extend-sdk-migrations.md new file mode 100644 index 000000000..62f4952b6 --- /dev/null +++ b/docs/06_extend-sdk-migrations.md @@ -0,0 +1,179 @@ +--- +id: extend-sdk-migration +title: "SDK Guide: Pipeline Element Migration" +sidebar_label: "SDK: PE Migration" +--- + +Pipeline element migrations allow you to automatically update and migrate both existing and pipeline elements when a new +version of StreamPipes is installed. This means that whenever you upgrade StreamPipes, all existing and future +pipeline elements will be directly compatible with the new version without any manual interaction. Pipeline elements +include adapters, data processors, and data sinks. + +:::info +Migrations will make their debut in StreamPipes version `0.93.0` and will be an integral part of the system going +forward. +However, it's important to note that this feature is not available in any of the previous versions of StreamPipes. To +take full advantage of migrations and their benefits, it is recommended to upgrade to version `0.93.0` or later. This +will +ensure that you have access to the latest enhancements and maintain compatibility with the evolving StreamPipes +platform. +::: + +## Define Migrations + +Whenever a pipeline element, be it an adapter, data processor, or data sink, undergoes changes that result in +modifications to its configuration options, developers must additionally create a migration procedure. This migration +process should be capable of smoothly transitioning all affected instances from the previous version to the new one. +The migration itself is automatically managed and executed by StreamPipes. Developers are only responsible for two key +aspects: + +* **Implementing the Concrete Migration**: Developers need to craft the specific migration logic that facilitates the + seamless transition of configuration options. +* **Registering the Migration**: Developers should register their migration procedures at the extensions service, + allowing StreamPipes to identify and apply the necessary updates to affected instances. + +By adhering to these two essential tasks, developers can ensure a hassle-free evolution of pipeline elements while +StreamPipes efficiently handles the orchestration of the migration process. + +The following gives a concrete example of creating a migration for +the [S7 adapter](./pe/org.apache.streampipes.connect.iiot.adapters.plc4x.s7.md). +Thereby, we assume this adapter has received a new input element which determines whether the connection should be made +authenticated or not. +This is represented by a simple boolean that is visualized as a toggle button in the UI. + +### Implementing a Concrete Migration + +StreamPipes offers three distinct migration mechanisms tailored to specific types of pipeline +elements: `AdapterMigrator`, `DataProcessorMigrator`, and `DataSinkMigrator`. +These migration mechanisms are presented as abstract classes and require the implementation of two fundamental methods: + +* `config()`: This method defines the configuration for the migration, encompassing all essential metadata related to + the migration process. +* `migrate()`: Within this method, the actual migration logic is to be implemented. It serves as the operational core + for facilitating the migration for the respective pipeline element. + +In accordance with the example described above, we will implement the `Plc4xS7AdapterMigrationV1` in the following. + +:::note +Before we begin, it's important to familiarize ourselves with two key conventions that guide our approach to migrations: + +* To maintain clarity and organization, all migration classes associated with a specific pipeline element are located + within a dedicated sub-package named `migration`. This sub-package is nested within the package of the respective + pipeline element. +* Migration classes are named according to a specific schema: `<elementClassName>MigrationV<targetVersion>`. For + example, if you are working on a migration for the PLC4x S7 adapter targeting version 1, the migration class would be + named `Plc4xS7AdapterMigrationV1`. +::: + +Let's begin with providing the migration's configuration: + +```java +@Override +public ModelMigratorConfig config() { + return new ModelMigratorConfig( + "org.apache.streampipes.connect.iiot.adapters.plc4x.s7", + SpServiceTagPrefix.ADAPTER, + 0, + 1 + ); +} +``` + +The migration config consists of the following four parts: + +* `targetAppId`: this needs to equal the app id of the targeted element +* `modelType`: the type of the element to be migrated, this can be one + of: `SpServiceTagPrefix.ADAPTER`, `SpServiceTagPrefix.DATA_PROCESSOR`, `SpServiceTagPrefix.DATA_SINK`. +* `fromVersion`: the version of the element that the migration expects as input +* `toVersion`: the version the element has after the migration (needs to be at least `fromVersion + 1`) + +The second step is to implement the actual migration logic. +In our example, we need to extend the existing static properties by an additional boolean property. + +```java +@Override +public MigrationResult<AdapterDescription> migrate(AdapterDescription element, IStaticPropertyExtractor extractor) throws RuntimeException { + + var config = element.getConfig(); + + var slideToggle = new SlideToggleStaticProperty(); + slideToggle.setDefaultValue(false); + slideToggle.setLabel("Authentication required?"); + config.add(slideToggle); + + element.setConfig(config); + return MigrationResult.success(element); +} +``` + +We've completed all the necessary steps for our migration. The final task remaining is to register the migration within +the service definition. + +### Registering the Migration + +Only when the migration is registered at the service definition, the migration is sent to the StreamPipes core service. +Therefore, we need to add the migration to the same service definition as the element to migrate. +In our example this is defined in `ConnectAdapterIiotInit`: + +```java jsx {22-24} showLineNumbers +@Override +public SpServiceDefinition provideServiceDefinition() { + return SpServiceDefinitionBuilder.create("connect-adapter-iiot", + "StreamPipes connect worker containing adapters relevant for the IIoT", + "", + 8001) + .registerAdapter(new MachineDataSimulatorAdapter()) + .registerAdapter(new FileReplayAdapter()) + .registerAdapter(new IfmAlMqttAdapter()) + .registerAdapter(new RosBridgeAdapter()) + .registerAdapter(new OpcUaAdapter()) + .registerAdapter(new Plc4xS7Adapter()) + .registerAdapter(new Plc4xModbusAdapter()) + .registerAdapter(new KafkaProtocol()) + .registerAdapter(new MqttProtocol()) + .registerAdapter(new NatsProtocol()) + .registerAdapter(new HttpStreamProtocol()) + .registerAdapter(new PulsarProtocol()) + .registerAdapter(new RocketMQProtocol()) + .registerAdapter(new HttpServerProtocol()) + .registerAdapter(new TubeMQProtocol()) + .registerMigrators( + new Plc4xS7AdapterMigrationV1() + ) + .build(); +``` + +<br /> + +## How Migrations are Handled Internally + +Migrations are handled by an interplay between the Extension Service, which provides the migrations, +and the StreamPipes Core Service, which manages the migrations, as shown in the figure below: +<img src="/img/06_sdk_migrations/migration-flow.png" alt="Interplay of extensions service and core to handle migrations"/> + +When an extensions service is initiated and has successfully registered itself with the core, it proceeds to send a +request to the core. This request includes a comprehensive list of all available migrations that have been registered +for it. Since this collection of migrations may encompass multiple migrations that affect the same pipeline element, +the migrations are first de-duplicated and then sorted based on their version range before being transmitted. + +Upon receiving these migrations, the core's actions can be categorized into two distinct parts: + +* Update descriptions for new elements +* Update descriptions for existing elements + +### Update Descriptions for New Elements + +Each migration transmitted from the extensions service to the core triggers the core to update the description of the +corresponding element stored in CouchDB. This is achieved by requesting the current configuration from the extensions +service and subsequently overwriting the existing configuration in the storage. + +### Update Descriptions for Existing Elements + +For each migration sent from the extensions service to the core, the core conducts a thorough check to determine if any +existing elements are affected by this migration. If such elements are identified, the extensions service is tasked with +requesting and subsequently executing the migration on behalf of the core. + +In scenarios where multiple applicable migrations exist for a single pipeline element, they are sequentially applied. +Success in this process allows the core to seamlessly update the configuration. However, if any issues arise, the +corresponding pipeline element is halted. In the case of processors and sinks, the associated pipeline is even marked +with a `needs attention` label, which comes apparent in the UI. diff --git a/website-v2/sidebars.json b/website-v2/sidebars.json index 336ba936a..61922703f 100644 --- a/website-v2/sidebars.json +++ b/website-v2/sidebars.json @@ -170,6 +170,7 @@ "extend-client", "extend-sdk-functions", "extend-sdk-event-model", + "extend-sdk-migration", "extend-sdk-stream-requirements", "extend-sdk-static-properties", "extend-sdk-output-strategies", diff --git a/website-v2/src/css/customTheme.css b/website-v2/src/css/customTheme.css index 72403971e..5b29da7e0 100644 --- a/website-v2/src/css/customTheme.css +++ b/website-v2/src/css/customTheme.css @@ -6,6 +6,9 @@ --ifm-color-primary-dark: #18125A; --ifm-color-primary-darker: #171155; --ifm-color-primary-darkest: #130E46; + --ifm-code-background: var(--color-primary-light); + --ifm-code-padding-horizontal: 0.2rem; + --ifm-code-padding-vertical: 0.12rem; --color-primary: rgb(27, 20, 100); --color-accent: rgb(57, 181, 74); @@ -17,6 +20,10 @@ --ifm-heading-font-family: 'Roboto'; } +code { + color: var(--color-accent); +} + .navbar__logo { height: 3rem; margin-left: 0.25rem; diff --git a/website-v2/static/img/06_sdk_migrations/migration-flow.png b/website-v2/static/img/06_sdk_migrations/migration-flow.png new file mode 100644 index 000000000..9b9ac6507 Binary files /dev/null and b/website-v2/static/img/06_sdk_migrations/migration-flow.png differ
