mcvsubbu commented on issue #2583: Create an abstraction for stream in realtime URL: https://github.com/apache/incubator-pinot/issues/2583#issuecomment-466807551 @EronWright somehow we completely missed this comment. We apologize for the late response. The re-factoring for offset not being a 'long' is hard and touches multiple places. The code on the server side (or that on the controller side) alone is probably easier than the idea of keeping it compatible between the two. We need to make sure that the segment completion works right with other types of offset. If the offset is a Comparable, then I think it could be a little easier. We just have to try out the re-factor. Alternatively, as you say, the high-level interface can be targeted. Two major issues we found with High level streams: 1. We are limited to consumption rate of one host. This was a non-starter for some of our high ingestion use cases. 2. Operationally it was a nightmare. One had to be careful about setting the Kafka instance config manually while expanding the cluster or replacing a node if it is down . We had gotten into situation where the Kafka traffic was split between a couple of hosts, leading to wrong results for the application. Addition of a new host to the cluster meant (sometimes) waiting for several days for that host to catch up, meanwhile making sure that new software is not deployed to that host. Some of these can be overcome with additional software, but we decided to go with LLC Hope this helps.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
