Hey Brian, Thanks for the links — they are exactly what we’ve been referring to.
It’s great to hear that 'in principle, it could still be compatible as far back as Prometheus 2.15.' If I’m interpreting this correctly, our initial thought should work, right? Specifically, simply changing the version from 3 to 2 in the index file for a V3-formatted TSDB block should make it recognizable and properly handled by Prometheus versions prior to 2.55, going back to 2.15. Of course, this is theoretical, and proper testing will be necessary once Prometheus releases a version that generates V3-formatted TSDB blocks. Bumping the version is absolutely justified given the breaking changes in the format. We are, however, seeking a feasible solution to meet our requirements — enabling rollback from Prometheus v3 to a version prior to v2.55 (though not as old as 2.15). Thanks! Best regards Gavine On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > The release notes for v2.55 describe this as an "upcoming index v3", and > links to > https://github.com/prometheus/prometheus/pull/14934 > which in turn links to > https://github.com/prometheus/prometheus/issues/14749 > > That has the full details of what's compatible. If I read this correctly, > the proposal is to remove some obsolete fields. Once the format change has > been made (and it hasn't yet), in principle it could still be compatible as > far back as prometheus 2.15. > > However, not only prometheus reads these chunk files (there's also thanos, > mimir, ...) so good engineering practice says that along with changing the > format, the version number will be bumped. > > On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: > >> Hi Brian, >> >> Thanks for your prompt reply. >> >> We were thinking the same — 'it's fairly obvious that won't work.' >> However, when we quickly reviewed the changes in v2.55.0, we noticed that >> it primarily makes FormatV3 recognizable while applying the same logic as >> FormatV2 (Backward compatibility with upcoming index v3 · >> prometheus/prometheus@5ccb069 · GitHub >> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >> ). >> Other changes under the tsdb folder seem irrelevant (though we could be >> mistaken, as we didn’t dive into the details). >> >> Additionally, Prometheus V2, starting from some versions, has already >> stopped reading/parsing Label Index 1...N and the Label Index Table — both >> of which are removed in Format V3. All this makes us wonder if simply >> changing the version number would suffice. >> >> Are there any other changes in v2.55.0 that we should inspect further? >> >> Another question: Have we decided which v3 version will make this change >> effective? The current v3 is still generating FormatV2 TSDB. >> Thanks! >> >> Best regards >> Gavine >> >> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >> >>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>> >>> Two steps rollback is also not supported ? Like rollback from v3.x to >>> v2.55 and then rollback from v2.55 to v2.53 without breaking the storage >>> format or storage compatibility? >>> >>> >>> That is exactly what Ben said. v2.55 knows how to work with the data >>> format from v3, but will not convert it back to the older format. >>> >>> It sounds to me like you would be best served by staying on the LTS >>> version v2.53, which still has bug and security fix support. Why do you >>> want to run anything later? >>> >>> If you *absolutely* need new features introduced in later versions, then >>> you will have to remain between versions v2.55 and v3.x - as there is no >>> rollback to anything earlier than v2.55 without losing data. But that's >>> fine, because if you *need* those new features then you wouldn't be able to >>> rollback anyway. Bear in mind that you will also have to keep upgrading to >>> all v3.x versions to get bugfixes and security fixes, until the next LTS >>> release comes along. >>> >>> If you just want to test the newer versions then do that on a >>> non-production system where you know you'll never need to roll back prior >>> to v2.55 >>> >>> Gavine Xue wrote: >>> > do you foresee any issues if we develop a conversion tool to change >>> the version number in index files back to 2 before a rollback? >>> >>> I think it's fairly obvious that won't work. If the *only* difference >>> between the data formats was the version number, then there would have been >>> no need for a change in the version number. The change in the version >>> number implies that the way that the data is stored has been changed in an >>> incompatible way. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/CAAGrqKHSzhfxigeU0sL%2B-zERoXQgzS%3Dx5ue1prtBt8YB89VyRw%40mail.gmail.com.
