@xiiang, Yes, it is useful to have a tool to change the offset of a consuming segment in case it errors out due to data in stream. I have added this as a todo in https://github.com/apache/incubator-pinot/issues/4035
There is limited number of things that can be done if data in incoming stream is bad. As designed, Pinot will try its best ti AVOID skipping data in incoming stream, so no matter which way you tell it, it will try its best to consume the data. Right now, the only automatic recovery available is to just ignore the alerts. Pinot will keep retrying every so often (depending on how often the realtime repair daemon runs on the controller) and eventually will move to a later offset when (if) the retention in the underlying stream makes the offset unavailable. If the underlying stream has infinite retention, it will never recover. The current behavior on encountering a bad message is that the segment marks itself as being in OFFLINE state. Since all replicas will encounter the bad message, they will all be marked OFFLINE in idealstate. The reailtime segment fixer looks for such segments and creates a new segment with the next seq number, but the same start offset (or, if that offset is not available anymore, then the earliest one available). I can think of one way to solve this issue more quickly than waiting for the event to be retained out of the stream. We can introduce an API on the controller to fix all segments in such state with the latest offset (or some specific offset -- but then the offset is on a per-partition basis), so that when the segment fixer upper comes around, it tries to create the new segment with the offset it finds in the OFFLINE segment. Race conditions need to be taken care of -- what if the segment fixer runs while this command is being run (on some other controller), etc. I can't think of an easy way to fix these races. We do depend on certain assumptions to reduce such races (e.g. it takes less than 10 minutes to commit a segment), etc. -Subbu On 2020/02/20 23:25:27, Fu Xiang <[email protected]> wrote: > Brought this up as some users are asking for this feature. E.g. to skip bad > data produced from upstream. > > For high level consumer, users can change table config to make consumer > offset reset policy to latest then delete Kafka consumer group and bounce > table/server to achieve this. > > For simple consumer, I don't see any easy way to handle this except for > modifying segment metadata content in zookeeper. > > I feel we need to provide the tooling to operate on them. > > -- > Xiang Fu > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
