mcvsubbu commented on issue #3717: kafka consume problem URL: https://github.com/apache/incubator-pinot/issues/3717#issuecomment-458369257 If the upstream is putting bad data and we are not able to create segments, we will (effectively) stop consuming. Why is pushing large volume a problem? We automatically adjust to the volume and make a segment sooner. The only thing I can think of is if in some production instance, capacity enhancement is needed but machines are not available immediately. In that case, the best option is to drop the (realtime) table, and re-create it. This works if offline flows are enabled in the corresponding offline table. If the feature is to skip certain (identified) offsets, or otherwise auto-detect bad data, that may be useful, I suppose. The only way to get out of a bad row ingested is to skip a segment altogether, something done manually.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
