I am not a developer of prometheus, just a satisfied user who sometimes shares some PromQL tips.
I do know that Ben Kochie is a major figure in the project, therefore I suggest you follow his advice, and the documented advice, which I would summarise as: "Don't do that". It is of course open source, and you are free to do what you like with it, but if you don't use it in the recommended way then you support yourself. I think that's fair. On Sunday, 2 February 2025 at 23:23:28 UTC Gavine Xue wrote: > Hi Brian, > > Hope you had a good weekend! > > I’d like to confirm the exact behavior when rolling back from *Prometheus > v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, such > as *v2.54.1*. > > The *v3 migration guide* states: > > "... you will only be able to downgrade to v2.55, not lower, without > losing your TSDB persistent data." > > This gives me the impression that *downgrading to v2.54.1 is still > possible*, but I’d appreciate clarification on the expected behavior: > > 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing > FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* > 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. Prometheus > v2.54.1 won't start at all?* > > From reviewing the source code, this block: > 🔗 *db.go#L1600-L1615 > <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* > suggests that *Prometheus may fail even though openBlocks() returns OK > with FormatV3 blocks put into corrupted*. Could you confirm if this is > the intended behavior? > Thanks! > > Best regards > Gavine > On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: > >> Hey Brian, >> >> Thanks for the links — they are exactly what we’ve been referring to. >> >> It’s great to hear that 'in principle, it could still be compatible as >> far back as Prometheus 2.15.' If I’m interpreting this correctly, our >> initial thought should work, right? Specifically, simply changing the >> version from 3 to 2 in the index file for a V3-formatted TSDB block should >> make it recognizable and properly handled by Prometheus versions prior to >> 2.55, going back to 2.15. Of course, this is theoretical, and proper >> testing will be necessary once Prometheus releases a version that generates >> V3-formatted TSDB blocks. >> >> Bumping the version is absolutely justified given the breaking changes in >> the format. We are, however, seeking a feasible solution to meet our >> requirements — enabling rollback from Prometheus v3 to a version prior to >> v2.55 (though not as old as 2.15). >> >> Thanks! >> Best regards >> Gavine >> >> On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < >> promethe...@googlegroups.com> wrote: >> >>> The release notes for v2.55 describe this as an "upcoming index v3", and >>> links to >>> https://github.com/prometheus/prometheus/pull/14934 >>> which in turn links to >>> https://github.com/prometheus/prometheus/issues/14749 >>> >>> That has the full details of what's compatible. If I read this >>> correctly, the proposal is to remove some obsolete fields. Once the format >>> change has been made (and it hasn't yet), in principle it could still be >>> compatible as far back as prometheus 2.15. >>> >>> However, not only prometheus reads these chunk files (there's also >>> thanos, mimir, ...) so good engineering practice says that along with >>> changing the format, the version number will be bumped. >>> >>> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >>> >>>> Hi Brian, >>>> >>>> Thanks for your prompt reply. >>>> >>>> We were thinking the same — 'it's fairly obvious that won't work.' >>>> However, when we quickly reviewed the changes in v2.55.0, we noticed that >>>> it primarily makes FormatV3 recognizable while applying the same logic as >>>> FormatV2 (Backward compatibility with upcoming index v3 · >>>> prometheus/prometheus@5ccb069 · GitHub >>>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>>> ). >>>> Other changes under the tsdb folder seem irrelevant (though we could be >>>> mistaken, as we didn’t dive into the details). >>>> >>>> Additionally, Prometheus V2, starting from some versions, has already >>>> stopped reading/parsing Label Index 1...N and the Label Index Table — both >>>> of which are removed in Format V3. All this makes us wonder if simply >>>> changing the version number would suffice. >>>> >>>> Are there any other changes in v2.55.0 that we should inspect further? >>>> >>>> Another question: Have we decided which v3 version will make this >>>> change effective? The current v3 is still generating FormatV2 TSDB. >>>> Thanks! >>>> >>>> Best regards >>>> Gavine >>>> >>>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>>> >>>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>>> >>>>> Two steps rollback is also not supported ? Like rollback from v3.x to >>>>> v2.55 and then rollback from v2.55 to v2.53 without breaking the storage >>>>> format or storage compatibility? >>>>> >>>>> >>>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>>> format from v3, but will not convert it back to the older format. >>>>> >>>>> It sounds to me like you would be best served by staying on the LTS >>>>> version v2.53, which still has bug and security fix support. Why do you >>>>> want to run anything later? >>>>> >>>>> If you *absolutely* need new features introduced in later versions, >>>>> then you will have to remain between versions v2.55 and v3.x - as there >>>>> is >>>>> no rollback to anything earlier than v2.55 without losing data. But >>>>> that's >>>>> fine, because if you *need* those new features then you wouldn't be able >>>>> to >>>>> rollback anyway. Bear in mind that you will also have to keep upgrading >>>>> to >>>>> all v3.x versions to get bugfixes and security fixes, until the next LTS >>>>> release comes along. >>>>> >>>>> If you just want to test the newer versions then do that on a >>>>> non-production system where you know you'll never need to roll back prior >>>>> to v2.55 >>>>> >>>>> Gavine Xue wrote: >>>>> > do you foresee any issues if we develop a conversion tool to change >>>>> the version number in index files back to 2 before a rollback? >>>>> >>>>> I think it's fairly obvious that won't work. If the *only* difference >>>>> between the data formats was the version number, then there would have >>>>> been >>>>> no need for a change in the version number. The change in the version >>>>> number implies that the way that the data is stored has been changed in >>>>> an >>>>> incompatible way. >>>>> >>>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> >> To view this discussion visit >>> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/154d512d-cc68-4273-9f65-c35d5a834fe7n%40googlegroups.com.
