[
https://issues.apache.org/jira/browse/NIFI-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alaksiej Ščarbaty updated NIFI-16310:
-------------------------------------
Description:
h2. Problem
A Controller Service created by a component's {{migrateProperties}} is removed
and recreated whenever the version of a versioned Process Group changes. During
that the service's state is lost.
A stateless service is unaffected by that. Whereas for stateful service it
causes a silent state loss: the replacement service is correctly configured and
enabled, but it holds no state from the removed controller service.
_Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
reported as local modifications when a flow is compared with its registered
version. That handling does not extend to applying a version change.
h2. Root cause
{{StandardControllerServiceFactory.create}} does not assign a versioned
component id to the service it creates.
{{StandardVersionedComponentSynchronizer}} generates a new versioned id from
the service's instance identifier for any controller services without a
versioned id. Then it removes every local controller service whose version id
is absent from the proposed snapshot. There is no exception for controller
services created in {{{}migrateProperties{}}}.
h2. Proposed change
The case to support first: a Process Group is upgraded to a version that
declares the Controller Service which was already created locally in
{{{}migrateProperties{}}}. The service must survive with its current state, and
the flow must end up with one controller service, not two.
# Synchronization must not delete a Controller Service that was created in
{{{}migrateProperties{}}}.
# This local Controller Service must be set with a versioned component id
assigned to that Controller Service in the versioned process group.
The matching of Controller Services should go through the component that
references them: if a component's Controller Service property points at a
migration-created service now, and the same property in the new version points
at a declared service of the same type in the same group, they must be treated
as the same service.
h3. Other matching strategies
Neither the identifier nor the versioned component id can be used to match
these two services.
Matching by class and property values work well only when controller service
created by {{migrateProperties}} is identical to a service in the versioned
group (see edge cases for properties drift). This matching strategy also won't
work if there are 2 identical controller services in the versioned group.
h2. Edge cases
This solution creates a lot of edge cases, most of which are caused by
Controller Service implementation/properties drift.
Normally a versioned process group author must ensure there is no properties
drift between a Controller Service created in {{migrateProperties}} and the
Service from versioned process group. Thuds I'm not sure if we should focus on
these edge cases.
Nonetheless, for each edge case I added an option on how I'd resolve it.
*New version does not declare the service, including a downgrade or revert to a
version predating it.* Keep the local service, unversioned. Currently it is
removed and then recreated empty by calling {{{}migrateProperties{}}}.
*New version deletes the child process group holding the service.* Since the
whole process group is deleted, the created controller service should be also
deleted.
The following cases are related to properties/Controller Services drift when
versioned Controller Services don't match local Controller Services created in
{{{}migrateProperties{}}}.
If that happens, we create the declared services from the versioned group, but
all components keep using local services. The differences are reported to the
user. The drift has to be resolved manually.
# *New version declares a service of a different implementation type than the
local service.*
# *Declared service has different property values* {*}than the local
service{*}{*}.{*}
# *Declared service is in a different process group from the local service.*
# *Multiple declared services are present instead of a single local service
which is referenced by several components.*
# *Declared service is present instead of several local services.*
# *Referencing component is absent from the new version, either deleted or
itself a migration-created service with no versioned component id.*
h2. Reproduction
Reproduced against 2.12.0-SNAPSHOT on main.
System tests:
[https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
{{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
{{migrateProperties}} creates a stateful Controller SErvice.
* {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
service is replaced by the one the new version declares;
*
{{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
- fails: the service is recreated under the same id with an empty store;
* {{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails at
its precondition, because the upgrade it performs first already removes the
service, leaving the downgrade path unverified.
Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
unaffected.
was:
h2. Problem
A Controller Service created by a component's {{migrateProperties}} is removed
and recreated whenever the version of a versioned Process Group changes. During
that the service's state is lost.
A stateless service is unaffected by that. Whereas for stateful service it
causes a silent state loss: the replacement service is correctly configured and
enabled, but it holds no state from the removed controller service.
_Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
reported as local modifications when a flow is compared with its registered
version. That handling does not extend to applying a version change.
h2. Root cause
{{StandardControllerServiceFactory.create}} does not assign a versioned
component id to the service it creates.
{{StandardVersionedComponentSynchronizer}} generates a new versioned id from
the service's instance identifier for any controller services without a
versioned id. Then it removes every local controller service whose version id
is absent from the proposed snapshot. There is no exception for controller
services created in {{{}migrateProperties{}}}.
h2. Proposed change
The case to support first: a Process Group is upgraded to a version that
declares the Controller Service which was already created locally in
{{{}migrateProperties{}}}. The service must survive with its current state, and
the flow must end up with one controller service, not two.
# Synchronization must not delete a Controller Service that was created in
{{{}migrateProperties{}}}.
# This local Controller Service must be set with a versioned component id
assigned to that Controller Service in the versioned process group.
The matching of Controller Services should go through the component that
references them: if a component's Controller Service property points at a
migration-created service now, and the same property in the new version points
at a declared service of the same type in the same group, they must be treated
as the same service.
Neither the identifier nor the versioned component id can be used to match
these two services.
h2. Edge cases
This solution creates a lot of edge cases, most of which are caused by
Controller Service implementation/properties drift.
Normally a versioned process group author must ensure there is no properties
drift between a Controller Service created in {{migrateProperties}} and the
Service from versioned process group. Thuds I'm not sure if we should focus on
these edge cases.
Nonetheless, for each edge case I added an option on how I'd resolve it.
*New version does not declare the service, including a downgrade or revert to a
version predating it.* Keep the local service, unversioned. Currently it is
removed and then recreated empty by calling {{{}migrateProperties{}}}.
*New version deletes the child process group holding the service.* Since the
whole process group is deleted, the created controller service should be also
deleted.
The following cases are related to properties/Controller Services drift when
versioned Controller Services don't match local Controller Services created in
{{{}migrateProperties{}}}.
If that happens, we create the declared services from the versioned group, but
all components keep using local services. The differences are reported to the
user. The drift has to be resolved manually.
# *New version declares a service of a different implementation type than the
local service.*
# *Declared service has different property values* {*}than the local
service{*}{*}.{*}
# *Declared service is in a different process group from the local service.*
# *Multiple declared services are present instead of a single local service
which is referenced by several components.*
# *Declared service is present instead of several local services.*
# *Referencing component is absent from the new version, either deleted or
itself a migration-created service with no versioned component id.*
h2. Reproduction
Reproduced against 2.12.0-SNAPSHOT on main.
System tests:
[https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
{{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
{{migrateProperties}} creates a stateful Controller SErvice.
* {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
service is replaced by the one the new version declares;
*
{{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
- fails: the service is recreated under the same id with an empty store;
* {{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails at
its precondition, because the upgrade it performs first already removes the
service, leaving the downgrade path unverified.
Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
unaffected.
> A controller service created by migrateProperties is removed on versioned
> group upgrade
> ---------------------------------------------------------------------------------------
>
> Key: NIFI-16310
> URL: https://issues.apache.org/jira/browse/NIFI-16310
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 2.11.0
> Reporter: Alaksiej Ščarbaty
> Assignee: Alaksiej Ščarbaty
> Priority: Major
>
> h2. Problem
> A Controller Service created by a component's {{migrateProperties}} is
> removed and recreated whenever the version of a versioned Process Group
> changes. During that the service's state is lost.
> A stateless service is unaffected by that. Whereas for stateful service it
> causes a silent state loss: the replacement service is correctly configured
> and enabled, but it holds no state from the removed controller service.
> _Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
> reported as local modifications when a flow is compared with its registered
> version. That handling does not extend to applying a version change.
> h2. Root cause
> {{StandardControllerServiceFactory.create}} does not assign a versioned
> component id to the service it creates.
> {{StandardVersionedComponentSynchronizer}} generates a new versioned id from
> the service's instance identifier for any controller services without a
> versioned id. Then it removes every local controller service whose version id
> is absent from the proposed snapshot. There is no exception for controller
> services created in {{{}migrateProperties{}}}.
> h2. Proposed change
> The case to support first: a Process Group is upgraded to a version that
> declares the Controller Service which was already created locally in
> {{{}migrateProperties{}}}. The service must survive with its current state,
> and the flow must end up with one controller service, not two.
> # Synchronization must not delete a Controller Service that was created in
> {{{}migrateProperties{}}}.
> # This local Controller Service must be set with a versioned component id
> assigned to that Controller Service in the versioned process group.
> The matching of Controller Services should go through the component that
> references them: if a component's Controller Service property points at a
> migration-created service now, and the same property in the new version
> points at a declared service of the same type in the same group, they must be
> treated as the same service.
> h3. Other matching strategies
> Neither the identifier nor the versioned component id can be used to match
> these two services.
> Matching by class and property values work well only when controller service
> created by {{migrateProperties}} is identical to a service in the versioned
> group (see edge cases for properties drift). This matching strategy also
> won't work if there are 2 identical controller services in the versioned
> group.
> h2. Edge cases
> This solution creates a lot of edge cases, most of which are caused by
> Controller Service implementation/properties drift.
> Normally a versioned process group author must ensure there is no properties
> drift between a Controller Service created in {{migrateProperties}} and the
> Service from versioned process group. Thuds I'm not sure if we should focus
> on these edge cases.
> Nonetheless, for each edge case I added an option on how I'd resolve it.
> *New version does not declare the service, including a downgrade or revert to
> a version predating it.* Keep the local service, unversioned. Currently it is
> removed and then recreated empty by calling {{{}migrateProperties{}}}.
> *New version deletes the child process group holding the service.* Since the
> whole process group is deleted, the created controller service should be also
> deleted.
> The following cases are related to properties/Controller Services drift when
> versioned Controller Services don't match local Controller Services created
> in {{{}migrateProperties{}}}.
> If that happens, we create the declared services from the versioned group,
> but all components keep using local services. The differences are reported to
> the user. The drift has to be resolved manually.
> # *New version declares a service of a different implementation type than
> the local service.*
> # *Declared service has different property values* {*}than the local
> service{*}{*}.{*}
> # *Declared service is in a different process group from the local service.*
> # *Multiple declared services are present instead of a single local service
> which is referenced by several components.*
> # *Declared service is present instead of several local services.*
> # *Referencing component is absent from the new version, either deleted or
> itself a migration-created service with no versioned component id.*
> h2. Reproduction
> Reproduced against 2.12.0-SNAPSHOT on main.
> System tests:
> [https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
> {{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
> {{migrateProperties}} creates a stateful Controller SErvice.
> * {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
> service is replaced by the one the new version declares;
> *
> {{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
> - fails: the service is recreated under the same id with an empty store;
> * {{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails at
> its precondition, because the upgrade it performs first already removes the
> service, leaving the downgrade path unverified.
> Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
> unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
