Hi Marek, I think that this is a good proposal. Thanks for driving this change.
Please go ahead and create a PIP to document this change and bring it to voting. You can see in https://github.com/apache/pulsar/tree/master/pip how proposals are documented in PIP documents. There's a template available. You'd create a PR with the PIP document and bring it to voting on this mailing list. After the vote has passed, the PIP PR will be merged and the reviews of the implementation PRs could follow. -Lari On 2024/04/16 12:57:42 Marek Czajkowski wrote: > Hello everyone, I'd like to initiate a discussion regarding the compaction > feature in Pulsar topics. Currently, there are two types of compactors > available: `TwoPhaseCompactor` and `StrategicTwoPhaseCompactor`. The latter > is specifically utilized for internal load balancing purposes and is not > employed for regular compaction of Pulsar topics. On the other hand, the > former can be configured via `CompactionServiceFactory` in the > `broker.conf`. > > I believe it could be advantageous to introduce another type of topic > compactor that operates based on event time. Such a compactor would have > the capability to maintain desired messages within the topic while > preserving the order expected by external applications. Although > applications may send messages with the current event time, variations in > network conditions or redeliveries could result in messages being stored in > the Pulsar topic in a different order than intended. Implementing event > time-based checks could mitigate this inconvenience. > > To address this, I've drafted a pull request with a sample implementation, > which can be found at: https://github.com/apache/pulsar/pull/22517 > > I would greatly appreciate any feedback or thoughts you may have on this > proposal. > > -- > > > *Marek Czajkowski* > > Backend developer > > Website: www.ds.pl > > Email: marek.czajkow...@ds.pl > > Mob: +48 698167319 > > Dynamic Solutions sp. z o.o. sp.k. > > ul. Poleska 45/1, 15-114 Białystok > > Your personal data controller is Dynamic Solutions Sp. z o.o. sp. k. > For GDPR compliance details, visit our website > <https://www.ds.pl/information-clause-email-correspondence.html>. >
