Do you mind creating a Github issue for this feature? - Sijie
On Tue, May 5, 2020 at 11:26 AM Patrick Hemmer <[email protected]> wrote: > Yes, it seems like keeping compactions around instead of deleting them > would be useful. Even better if their usage is automatic when the consumer > sets the 'read compacted' flag. Though a manual retrieval of the specific > compaction shouldn't be too difficult. > > -Patrick > > > On Tue, May 5, 2020, at 14:01, Sijie Guo wrote: > > Hi Patrick, > > Currently Pulsar only supports one compaction state per topic at this > moment. However the underneath implementation doesn't prevent supporting > multiple concurrent compactions as we just need to track the complications > in different cursors. As I understand here, you want the ability to keep > the compaction state as a snapshot hence you can go back in time to lookup > the compaction data. Is that the fair statement? > > - sijie > > On Mon, May 4, 2020 at 5:20 PM Patrick Hemmer <[email protected]> > wrote: > > > I'm looking to use Pulsar to store some timeseries data with infinite > retention. For a simplistic example, lets say it's a bunch of orders > lifecycle events (created, shipped, received, cancelled, etc). I want to be > able to retrieve the state of the open orders at a given point in time, and > then all order events up to a second point in time. > > At first compaction seems like what I want here, as with the first point > in time, I don't care about orders in the past which are closed. So > compaction would keep me from having to replay all events from the > beginning of time. However as I want to be able to retrieve any period of > time, I would need multiple compaction points so I could use the one > closest to my start time. > > Is this possible? > From my understanding reading through the documentation ( > https://pulsar.apache.org/docs/en/concepts-topic-compaction/), it looks > like there's only a single compaction point. So if I performed a compaction > today, but I wanted to retrieve last month's data, I'd have to replay from > the beginning of time. > Can Pulsar handle this, or will I have to create some manual method of > snapshotting and storing the state at periodic intervals? > > Thanks > > -Patrick > > >
