[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2854: -- Description: The flowfile, swapfile, provenance, and content repositories play a very important roll in NiFi's ability to be safely upgraded and rolled back. We need to have well documented behaviors, designs, and version adherence so that users can safely rely on these mechanisms. Once this is formalized and in place we should update our versioning guidance to reflect this as well. The following would be true from NiFi 1.2.0 onward * No changes to how the repositories are persisted to disk can be made which will break forward/backward compatibility and specifically this means that things like the way each is serialized to disk cannot change. * If changes are made which impact forward or backward compatibility they should be reserved for major releases only and should include a utility to help users with pre-existing data convert from some older format to the newer format. It may not be feasible to have rollback on major releases. * The content repository should not be changed within a major release cycle in any way that will harm forward or backward compatibility. * The flow file repository can change in that new fields can be added to existing write ahead log record types but no fields can be removed nor can any new types be added. Once a field is considered required it must remain required. Changes may only be made across minor version changes - not incremental. * Swap File storage should follow very similar rules to the flow file repository. Adding a schema to the swap file header may allow some variation there but the variation should only be hints to optimize how they're processed and not change their behavior otherwise. Changes are only permitted during minor version releases. * Provenance repository changes are only permitted during minor version releases. These changes may include adding or removing fields from existing event types. If a field is considered required it must always be considered required. If a field is removed then it must not be a required field and there must be a sensible default an older version could use if that value is not found in new data once rolled back. New event types may be added. Fields or event types not known to older version, if seen after a rollback, will simply be ignored. The following also would be true: * Apache NiFi 1.0.0 repositories should work just fine when applied to an Apache NiFi 1.1.0 installation. * Repositories made/updated in Apache NiFi 1.1.0 onward would not work in older Apache NiFi releases (such as 1.0.0) was: The flowfile, swapfile, provenance, and content repositories play a very important roll in NiFi's ability to be safely upgraded and rolled back. We need to have well documented behaviors, designs, and version adherence so that users can safely rely on these mechanisms. Once this is formalized and in place we should update our versioning guidance to reflect this as well. The following would be true from NiFi 1.2.0 onward * No changes to how the repositories are persisted to disk can be made which will break forward/backward compatibility and specifically this means that things like the way each is serialized to disk cannot change. * If changes are made which impact forward or backward compatibility they should be reserved for major releases only and should include a utility to help users with pre-existing data convert from some older format to the newer format. It may not be feasible to have rollback on major releases. * The content repository should not be changed within a major release cycle in any way that will harm forward or backward compatibility. * The flow file repository can change in that new fields can be added to existing write ahead log record types but no fields can be removed nor can any new types be added. Once a field is considered required it must remain required. Changes may only be made across minor version changes - not incremental. * Swap File storage should follow very similar rules to the flow file repository. Adding a schema to the swap file header may allow some variation there but the variation should only be hints to optimize how they're processed and not change their behavior otherwise. Changes are only permitted during minor version releases. * Provenance repository changes are only permitted during minor version releases. These changes may include adding or removing fields from existing event types. If a field is considered required it must always be considered required. If a field is removed then it must not be a required field and there must be a sensible default an older version could use if that value is not found in new data once rolled back. New event types may be added. Fields or event types not known to older version, if seen aft
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2854: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2854: - Fix Version/s: (was: 1.2.0) 1.1.0 > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2854: - Status: Patch Available (was: Open) > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2854: -- Description: The flowfile, swapfile, provenance, and content repositories play a very important roll in NiFi's ability to be safely upgraded and rolled back. We need to have well documented behaviors, designs, and version adherence so that users can safely rely on these mechanisms. Once this is formalized and in place we should update our versioning guidance to reflect this as well. The following would be true from NiFi 1.2.0 onward * No changes to how the repositories are persisted to disk can be made which will break forward/backward compatibility and specifically this means that things like the way each is serialized to disk cannot change. * If changes are made which impact forward or backward compatibility they should be reserved for major releases only and should include a utility to help users with pre-existing data convert from some older format to the newer format. It may not be feasible to have rollback on major releases. * The content repository should not be changed within a major release cycle in any way that will harm forward or backward compatibility. * The flow file repository can change in that new fields can be added to existing write ahead log record types but no fields can be removed nor can any new types be added. Once a field is considered required it must remain required. Changes may only be made across minor version changes - not incremental. * Swap File storage should follow very similar rules to the flow file repository. Adding a schema to the swap file header may allow some variation there but the variation should only be hints to optimize how they're processed and not change their behavior otherwise. Changes are only permitted during minor version releases. * Provenance repository changes are only permitted during minor version releases. These changes may include adding or removing fields from existing event types. If a field is considered required it must always be considered required. If a field is removed then it must not be a required field and there must be a sensible default an older version could use if that value is not found in new data once rolled back. New event types may be added. Fields or event types not known to older version, if seen after a rollback, will simply be ignored. was:Currently, if we start up a new version of NiFi and the format of the Data Provenance data has changed, we are not able to roll back to a previous version of NiFi. If we do, NiFi will fail to read the Provenance Data and not start up. We should instead provide the ability to write data to the repository in such a way that old versions of the repository will still be able to read the data, so that we can roll back > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but t
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2854: -- Summary: Enable repositories to support upgrades and rollback in well defined scenarios (was: Allow Provenance Repository to roll back to a previous implementation) > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > Currently, if we start up a new version of NiFi and the format of the Data > Provenance data has changed, we are not able to roll back to a previous > version of NiFi. If we do, NiFi will fail to read the Provenance Data and not > start up. We should instead provide the ability to write data to the > repository in such a way that old versions of the repository will still be > able to read the data, so that we can roll back -- This message was sent by Atlassian JIRA (v6.3.4#6332)