I think your problem has been solved in Prometheus v2.53.5: [ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762 <https://github.com/prometheus/prometheus/pull/16762>
On Tuesday, 4 February 2025 at 01:10:46 UTC Gavine Xue wrote: > Thanks, Brian, for the heads-up. > > Hi Ben, > > Could you please help clarify the following questions? > > 1. > > Regarding the statement in the V3 migration guide: *"... you will only > be able to downgrade to v2.55, not lower, without losing your TSDB > persistent data." *— does this mean that downgrading from v3.x to a > pre-v2.55.0 version is completely unsupported, or is it technically > possible but would result in the loss of FormatV3 data after rollback? > This > distinction is crucial for our customers. > 2. > > Is there a version — e.g., in a dev branch — where this change is > already effective (i.e., generating FormatV3 TSDB data) so that we can > test > it? > > Thanks! > > Best regards > Gavine > > On Monday, 3 February 2025 at 7:56:41 pm UTC+11 Brian Candler wrote: > >> I am not a developer of prometheus, just a satisfied user who sometimes >> shares some PromQL tips. >> >> I do know that Ben Kochie is a major figure in the project, therefore I >> suggest you follow his advice, and the documented advice, which I would >> summarise as: "Don't do that". >> >> It is of course open source, and you are free to do what you like with >> it, but if you don't use it in the recommended way then you support >> yourself. I think that's fair. >> >> On Sunday, 2 February 2025 at 23:23:28 UTC Gavine Xue wrote: >> >>> Hi Brian, >>> >>> Hope you had a good weekend! >>> >>> I’d like to confirm the exact behavior when rolling back from *Prometheus >>> v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, >>> such as *v2.54.1*. >>> >>> The *v3 migration guide* states: >>> >>> "... you will only be able to downgrade to v2.55, not lower, without >>> losing your TSDB persistent data." >>> >>> This gives me the impression that *downgrading to v2.54.1 is still >>> possible*, but I’d appreciate clarification on the expected behavior: >>> >>> 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing >>> FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* >>> 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. >>> Prometheus v2.54.1 won't start at all?* >>> >>> From reviewing the source code, this block: >>> 🔗 *db.go#L1600-L1615 >>> <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* >>> suggests that *Prometheus may fail even though openBlocks() returns OK >>> with FormatV3 blocks put into corrupted*. Could you confirm if this is >>> the intended behavior? >>> Thanks! >>> >>> Best regards >>> Gavine >>> On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: >>> >>>> Hey Brian, >>>> >>>> Thanks for the links — they are exactly what we’ve been referring to. >>>> >>>> It’s great to hear that 'in principle, it could still be compatible as >>>> far back as Prometheus 2.15.' If I’m interpreting this correctly, our >>>> initial thought should work, right? Specifically, simply changing the >>>> version from 3 to 2 in the index file for a V3-formatted TSDB block should >>>> make it recognizable and properly handled by Prometheus versions prior to >>>> 2.55, going back to 2.15. Of course, this is theoretical, and proper >>>> testing will be necessary once Prometheus releases a version that >>>> generates >>>> V3-formatted TSDB blocks. >>>> >>>> Bumping the version is absolutely justified given the breaking changes >>>> in the format. We are, however, seeking a feasible solution to meet our >>>> requirements — enabling rollback from Prometheus v3 to a version prior to >>>> v2.55 (though not as old as 2.15). >>>> >>>> Thanks! >>>> Best regards >>>> Gavine >>>> >>>> On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < >>>> promethe...@googlegroups.com> wrote: >>>> >>>>> The release notes for v2.55 describe this as an "upcoming index v3", >>>>> and links to >>>>> https://github.com/prometheus/prometheus/pull/14934 >>>>> which in turn links to >>>>> https://github.com/prometheus/prometheus/issues/14749 >>>>> >>>>> That has the full details of what's compatible. If I read this >>>>> correctly, the proposal is to remove some obsolete fields. Once the >>>>> format >>>>> change has been made (and it hasn't yet), in principle it could still be >>>>> compatible as far back as prometheus 2.15. >>>>> >>>>> However, not only prometheus reads these chunk files (there's also >>>>> thanos, mimir, ...) so good engineering practice says that along with >>>>> changing the format, the version number will be bumped. >>>>> >>>>> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >>>>> >>>>>> Hi Brian, >>>>>> >>>>>> Thanks for your prompt reply. >>>>>> >>>>>> We were thinking the same — 'it's fairly obvious that won't work.' >>>>>> However, when we quickly reviewed the changes in v2.55.0, we noticed >>>>>> that >>>>>> it primarily makes FormatV3 recognizable while applying the same logic >>>>>> as >>>>>> FormatV2 (Backward compatibility with upcoming index v3 · >>>>>> prometheus/prometheus@5ccb069 · GitHub >>>>>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>>>>> ). >>>>>> Other changes under the tsdb folder seem irrelevant (though we could be >>>>>> mistaken, as we didn’t dive into the details). >>>>>> >>>>>> Additionally, Prometheus V2, starting from some versions, has already >>>>>> stopped reading/parsing Label Index 1...N and the Label Index Table — >>>>>> both >>>>>> of which are removed in Format V3. All this makes us wonder if simply >>>>>> changing the version number would suffice. >>>>>> >>>>>> Are there any other changes in v2.55.0 that we should inspect further? >>>>>> >>>>>> Another question: Have we decided which v3 version will make this >>>>>> change effective? The current v3 is still generating FormatV2 TSDB. >>>>>> Thanks! >>>>>> >>>>>> Best regards >>>>>> Gavine >>>>>> >>>>>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>>>>> >>>>>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>>>>> >>>>>>> Two steps rollback is also not supported ? Like rollback from v3.x >>>>>>> to v2.55 and then rollback from v2.55 to v2.53 without breaking the >>>>>>> storage >>>>>>> format or storage compatibility? >>>>>>> >>>>>>> >>>>>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>>>>> format from v3, but will not convert it back to the older format. >>>>>>> >>>>>>> It sounds to me like you would be best served by staying on the LTS >>>>>>> version v2.53, which still has bug and security fix support. Why do >>>>>>> you >>>>>>> want to run anything later? >>>>>>> >>>>>>> If you *absolutely* need new features introduced in later versions, >>>>>>> then you will have to remain between versions v2.55 and v3.x - as there >>>>>>> is >>>>>>> no rollback to anything earlier than v2.55 without losing data. But >>>>>>> that's >>>>>>> fine, because if you *need* those new features then you wouldn't be >>>>>>> able to >>>>>>> rollback anyway. Bear in mind that you will also have to keep >>>>>>> upgrading to >>>>>>> all v3.x versions to get bugfixes and security fixes, until the next >>>>>>> LTS >>>>>>> release comes along. >>>>>>> >>>>>>> If you just want to test the newer versions then do that on a >>>>>>> non-production system where you know you'll never need to roll back >>>>>>> prior >>>>>>> to v2.55 >>>>>>> >>>>>>> Gavine Xue wrote: >>>>>>> > do you foresee any issues if we develop a conversion tool to >>>>>>> change the version number in index files back to 2 before a rollback? >>>>>>> >>>>>>> I think it's fairly obvious that won't work. If the *only* >>>>>>> difference between the data formats was the version number, then there >>>>>>> would have been no need for a change in the version number. The change >>>>>>> in >>>>>>> the version number implies that the way that the data is stored has >>>>>>> been >>>>>>> changed in an incompatible way. >>>>>>> >>>>>> -- >>>>> >>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to prometheus-use...@googlegroups.com. >>>>> >>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/9b1fddc4-a260-479a-be68-4ab2e1aa6537n%40googlegroups.com.
