navina opened a new issue #7988: URL: https://github.com/apache/pinot/issues/7988
There are streaming systems which require additional client-initiated operations to guarantee the availability of data for replay. For example, Google Pubsub does **not** expose any notion of `offsets` for messages. It only delivers the messages that were published after a consumer is created. In order to replay or rewind, the client has to manually create/manage `snapshots` (aka checkpoints), which serves as an alternate mechanism to replay messages from a particular point in the stream. Pinot's realtime streaming ingestion is very closely modeled to kafka's semantics. In order to accommodate other streaming systems, it would be useful to expand the existing `PartitionGroupConsumer`. Proposal is to provide a mechanism to `commit` the state to the streaming source. For details on the proposal, please see - https://docs.google.com/document/d/17eywG2zYHv72ye3nBwVSdxlF7JIcTlqcNkov2TCqaGs/edit?usp=sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
