This isn’t a feature in any normal sense? It’s a commitment, and it requires every contributor to consider it as part of work they produce. I agree we should put in some work to make this easier, and to provide some test guarantees that we don’t break it. But are we overstating the difficulty? We already use the descriptor version for serialising the contents, it’s only the metadata that ignores this today - and downgrade tests should be as simple as making upgrade tests go backwards.
But in the meantime none of that seems like an excuse to fail to address this in work that may break it, where it is not particularly costly to do so.
On 21 Feb 2023, at 08:57, Branimir Lambov <branimir.lam...@datastax.com> wrote:
It appears to me that the first thing we need to start this feature off is a definition of a suite of tests together with a set of rules to keep the suite up to date with new features as they are introduced. The moment that suite is in place, we can start having some confidence that we can enforce downgradability.
Something like this will definitely catch incompatibilities in SSTable formats (such as the one in CASSANDRA-17698 that I managed to miss during review), but will also be able to identify incompatible system schema changes among others, and at the same time rightfully ignore non-breaking changes such as modifications to the key cache serialization formats.
Is downgradability in scope for 5.0? It is a feature like any other, and I don't see any difficulty adding it (with support for downgrade to 4.x) a little later in the 5.x timeline.
Regards, Branimir I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce multiple sstable formats support. It allows for providing an implementation of SSTableFormat along with SSTableReader and SSTableWriter. That could be extended easily to support different implementations for certain version ranges, like one impl for ma-nz, other for oa+, etc. without having a confusing implementation with a lot of conditional blocks. Old formats in such case could be maintained separately from the main code and easily switched any time.
thanks
- - -- --- ----- -------- -------------
Jacek Lewandowski
Hi,
Upgrading the cluster can be painful for some orgs with mission critical Cassandra cluster, where they cannot tolerate less availability because of the inability to replace the downed node. They also need to plan rolling back to the previous state when something happens along the way. The change I proposed in CASSANDRA-8110 is to achieve the goal of at least enabling SSTable streaming during the upgrade by not upgrading the SSTable version. This can make the cluster to easily rollback to the previous version.
Downgrading SSTable is not the primary focus (though Cassandra needs to implement the way to write SSTable in older versions, so it is somewhat related.)
I'm preparing the design doc for the change. Also, if I should create a separate ticket from CASSANDRA-8110 for the clarity of the goal of the change, please let me know.
FWIW I think 8110 is the right approach, even if it isn’t a panacea. We will have to eventually also tackle system schema changes (probably not hard), and may have to think a little carefully about other things, eg with TTLs the format change is only the contract about what values can be present, so we have to make sure the data validity checks are consistent with the format we write. It isn’t as simple as writing an earlier version in this case (unless we permit truncating the TTL, perhaps)
In a self-organising community, things that aren’t self-policed naturally end up policed in an adhoc manner, and with difficulty. I’m not sure that’s the same as arbitrary enforcement. It seems to me the real issue is nobody noticed this was agreed and/or forgot and didn’t think about it much.
But, even without any prior agreement, it’s perfectly reasonable to request that things do not break compatibility if they do not need to, as part of the normal patch integration process.
Issues with 3.1->4.0 aren’t particularly relevant as they predate any agreement to do this. But we can and should address the problem of new columns in schema tables, as this happens often between versions. I’m not sure it has in 4.1 though?
Regarding downgrade versions, surely this should simply be the same as upgrade versions we support?
I'm not even convinced even 8110 addresses this - just writing sstables in old versions won't help if we ever add things like new types or new types of collections without other control abilities. Claude's other email in another thread a few hours ago talks about some of these surprises - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added to system/local. This means that 3.1 system can not read the table as it has no definition for it. I tried marking the column for deletion in the metadata and in the serialization header. The later got past the column not found problem, but I suspect that it just means that data columns after broadcast_port shifted and so incorrectly read." - this is a harder problem to solve than just versioning sstables and network protocols.
Stepping back a bit, we have downgrade ability listed as a goal, but it's not (as far as I can tell) universally enforced, nor is it clear at which point we will be able to concretely say "this release can be downgraded to X". Until we actually define and agree that this is a real goal with a concrete version where downgrade-ability becomes real, it feels like things are somewhat arbitrarily enforced, which is probably very frustrating for people trying to commit work/tickets.
- Jeff
I’m a big fan of maintaining backward compatibility. Downgradability implies that we could potentially roll back an upgrade at any time. While I don’t think we need to retain the ability to downgrade in perpetuity it would be a good objective to maintain strict backward compatibility and therefore downgradability until a certain point. This would imply versioning metadata and extending it in such a way that prior version(s) could continue functioning. This can certainly be expensive to implement and might bloat on-disk storage. However, we could always offer an option for the operator to optimize the on-disk structures for the current version then we can rewrite them in the latest version. This optimizes the storage and opens up new functionality. This means new features that can work with old on-disk structures will be available while others that strictly require new versions of the data structures will be unavailable until the operator migrates to the new version. This migration IMO should be irreversible. Beyond this point the operator will lose the ability to downgrade which is ok. Dinesh There has been progress on
Which is similar to what datastax does for DSE. Would this be an acceptable solution?
Jake It seems “ An alternative solution is to implement/complete CASSANDRA-8110” can give us more options if it is finished😉 Hi everyone,
There has been a discussion lately about changes to the sstable format in the context of being able to abort a cluster upgrade, and the fact that changes to sstables can prevent downgraded nodes from reading any data written during their temporary operation with the new version.
Downgradability is a worthy goal and is listed in the current roadmap. I would like to open a discussion here on how it would be achieved.
My understanding of what has been suggested so far translates to: - avoid changes to sstable formats; - if there are changes, implement them in a way that is backwards-compatible, e.g. by duplicating data, so that a new version is presented in a component or portion of a component that legacy nodes will not try to read; - if the latter is not feasible, make sure the changes are only applied if a feature flag has been enabled.
To me this approach introduces several risks: - it bloats file and parsing complexity; - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF ticket once this requirement is in place); - it needs care to avoid risky solutions to address technical issues with the format versioning (e.g. staying on n-versions for 5.0 and needing a bump for a 4.1 bugfix might require porting over support for new features);
- it requires separate and uncoordinated solutions to the problem and switching mechanisms for each individual change.
An alternative solution is to implement/complete CASSANDRA-8110, which provides a method of writing sstables for a target version. During upgrades, a node could be set to produce sstables corresponding to the older version, and there is a very straightforward way to implement modifications to formats like the tickets above to conform to its requirements.
What do people think should be the way forward?
Regards,
you are the apple of my eye !
--
--
|
