mcvsubbu commented on issue #3717: kafka consume problem URL: https://github.com/apache/incubator-pinot/issues/3717#issuecomment-458632932 @ly923976094 I assume you don't have any redundancy on your servers and therefore you cannot restart it. Note that in this case you may not be able to update to newer software since that will involve a restart. @kishoreg I assume that if data pushed to Kafka has bad values, then we eventually want to fix and restart consumption. In this use case, I think having a feature to ignore segments (until some offset or time value, perhaps) is valuable. As soon as we detect there is bad data, we can set Pinot to ignore segments until some criteria (if offsets, it will need to be a vector of offsets for partitions). Such a feature can also work for cases where data is invalid (in that we are not able to make a segment, for example). I am just trying to get a use case so as to define the feature we want as best as we can. Assuming we provide a pause/restart feature, the following questions pop up: * Do we want to pause on a per-partition basis? I assume yes. * Do we care that we may lose data (due to Kafka retention) and restart from wherever earliest we can? * When we pause, do we want to discard the current consuming segment, or keep until consumed (perhaps the latter is beneficial for the use case that @ly923976094 has described, but maybe the former for the use case that you describe). The feature sounds great, I am just trying to determine a use case for it and define it so that it satisfies (at least) that particular use case.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
