anuragrai16 commented on PR #19529: URL: https://github.com/apache/pinot/pull/19529#issuecomment-5644479699
> If I understand right, you are emitting a metric (of value 1) with the current decoder class name. Is this not an overkill? Can you just log the decoder class name? Even log at some regular cadence if needed? > > I saw a proposal float by for pinot internal table. Perhaps you could add a row/column in that table? Hi @mcvsubbu - Thanks for checking the diff. We're trying to migrate 1000s of table decoders on live production tables, each of which consumes from Kafka topics upto 256 partitions, while Pinot has no observability into what decoder was used for a mutable segment, or being used for a consuming segment. A log is ephemeral and not a reliable way to solve this problem. At the cost/overkill part, we are just emitting one additional gauge per consuming segment, using the existing setValueOfGauge register-once path, no new gauge objects created per tick, just an AtomicLong.set(1). The overhead is negligible compared to what each consuming segment already emits. As for the proposal for Pinot internal table, it is a larger effort that's still in design. This gauge is a minimal, backward-compatible addition (constant 1 emitted alongside the already-existing `LLC_PARTITION_CONSUMING` gauge on the same code path, same register-once/AtomicLong mechanism) that solves the immediate operational need without waiting on new infrastructure. Let me know your thoughts, I feel this metadata belongs in the segment as a first class citizen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
