EronWright commented on issue #2583: Create an abstraction for stream in realtime URL: https://github.com/apache/incubator-pinot/issues/2583#issuecomment-442703012 I took a look at the abstractions with en eye to how to map a [dynamic sharding scheme](http://pravega.io/docs/latest/pravega-concepts/#stream-segments) (e.g. that of Pravega) into the partition level consumer interface of Pinot. One idea I had was to develop some notion of virtual partitions (e.g. fixed partitioning of the stream's keyspace). The per-partition checkpoint requirement could be satisfied, but the notion of an offset may be a problem. How central is the notion of offset, could it be factored out as an implementation detail of the consumer? Alternatively one could target the high-level stream consumer interface. I don't understand what the tradeoff would be. I also don't understand how checkpoints work in the high-level API; does Pinot expect a global checkpoint or a per-indexer checkpoint? Thanks for any insights!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
