Hi Patrick, Currently Pulsar only supports one compaction state per topic at this moment. However the underneath implementation doesn't prevent supporting multiple concurrent compactions as we just need to track the complications in different cursors. As I understand here, you want the ability to keep the compaction state as a snapshot hence you can go back in time to lookup the compaction data. Is that the fair statement?
- sijie On Mon, May 4, 2020 at 5:20 PM Patrick Hemmer <[email protected]> wrote: > I'm looking to use Pulsar to store some timeseries data with infinite > retention. For a simplistic example, lets say it's a bunch of orders > lifecycle events (created, shipped, received, cancelled, etc). I want to be > able to retrieve the state of the open orders at a given point in time, and > then all order events up to a second point in time. > > At first compaction seems like what I want here, as with the first point > in time, I don't care about orders in the past which are closed. So > compaction would keep me from having to replay all events from the > beginning of time. However as I want to be able to retrieve any period of > time, I would need multiple compaction points so I could use the one > closest to my start time. > > Is this possible? > From my understanding reading through the documentation ( > https://pulsar.apache.org/docs/en/concepts-topic-compaction/), it looks > like there's only a single compaction point. So if I performed a compaction > today, but I wanted to retrieve last month's data, I'd have to replay from > the beginning of time. > Can Pulsar handle this, or will I have to create some manual method of > snapshotting and storing the state at periodic intervals? > > Thanks > > -Patrick >
