Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Patrik, Ah, I guess that was an important distinction! Thanks for clarifying. If there's a method to call on the RocksDB, then you probably want to consider just registering the metric as a Gauge that delegates to RocksDB to get the value, as opposed to the normal pattern of using a Sensor with sampled stats metrics. Thanks, -John On Mon, Jul 15, 2019 at 1:05 PM Patrik Kleindl wrote: > > Hi John > Translated it wrong, I meant ‚might‘ instead of ‚may‘. > If I find the time I‘ll take a look at the code how they could be added as > metrics. > Thanks for your input. > Regards > Patrik > > > Am 15.07.2019 um 18:03 schrieb John Roesler : > > > > Hey Patrik, > > > > Since KIP-471 is already accepted, and since this idea is not a > > trivial extension of the KIP, I think we'd need to do a new KIP. > > > > Some points to consider: these additions could not be made to the > > KeyValueStore interface, since they'd be only applicable to > > RocksDB-backed stores, but there is no RocksDBKeyValueStore, > > which could be used to query the property. Rather, the rocks interface > > is wrapped inside multiple layers that comprise the KeyValueStore. > > Since these layers themselves are not public interfaces, it would be a > > challenge to think of how to get access to these methods from > > user-space code. > > > > On the other hand, could we just add new metrics to access those > > properties? Offhand, it's unclear to me why you said they "may not be > > exposed". > > > > Thanks, > > -John > > > >> On Mon, Jul 15, 2019 at 5:11 AM Patrik Kleindl wrote: > >> > >> Hi > >> Adding this here because I asked and may have found the answer: > >> The memory consumption may not be exposed as RocksDB metrics, but they > >> should be available as properties of the RocksDB instance itself. > >> RocksDBStore could easily make this available by allowing access > >> to db.getProperty. > >> Available data (examples): > >> rocksdb.block-cache-usage > >> rocksdb.estimate-table-readers-mem > >> rocksdb.cur-size-all-mem-tables > >> > >> Reference: > >> https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB and > >> found the names of the properties here > >> https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h > >> > >> Would it make sense to allow this kind of access? > >> Should this be part of a follow-up KIP? > >> > >> best regards > >> > >> Patrik > >> > >>> On Wed, 19 Jun 2019 at 21:55, Bill Bejeck wrote: > >>> > >>> Hi Bruno, > >>> > >>> Just getting caught up on this KIP thread. Looks good to me, and I don't > >>> have any additional comments to what's already been presented. > >>> > >>> Thanks, > >>> Bill > >>> > On Wed, Jun 19, 2019 at 1:42 PM Bruno Cadonna wrote: > > John and Guozhang, > > thank you for your comments. > > @Guozhang could you please also vote on the voting thread so that we > have all votes in one place. > > @John, the only situation I can think of where a non-uniform > configuration of segments would make sense is to account for > seasonality. But this would be a really advanced configuration IMO. > > Best, > Bruno > > > On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > > > One last thought. I think it makes sense what you propose for merging > > the metrics when a logical store is composed of multiple physical > > stores. > > > > The basic standard for these metrics is that they should be relevant > > to performance, and they should be controllable via configurations, > > specifically via RocksDBConfigSetter. The config setter takes as input > > the store name. For Key/Value stores, this name is visible in the tags > > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > > metric rocksdb-state-id=STORE_X is showing a low block cache hit > > ratio, so I add a condition in my config setter to increase the block > > cache size for stores named STORE_X. > > > > For this case where the logical store has multiple physical stores, > > we're really talking about segmented stores, window stores and > > segmented stores. In these stores, every segment has a different name > > (logicalStoreName + "." + segmentId * segmentInterval), so the config > > setter would need to use a prefix match with respect to the metric tag > > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > > doable. > > > > It's also perfectly reasonable, since segments get rotated out all the > > time, it's implausible that you'd ever want a non-uniform > > configuration over the segments in a store. For this reason, > > specifically, it makes more sense just to summarize the metrics than > > to present them individually. Might be worth documenting the > > relationship, though. > > > > Thanks again, I'll vote now, > > -John > > > > On Wed, Jun 19, 2019 at 12:03 PM
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi John Translated it wrong, I meant ‚might‘ instead of ‚may‘. If I find the time I‘ll take a look at the code how they could be added as metrics. Thanks for your input. Regards Patrik > Am 15.07.2019 um 18:03 schrieb John Roesler : > > Hey Patrik, > > Since KIP-471 is already accepted, and since this idea is not a > trivial extension of the KIP, I think we'd need to do a new KIP. > > Some points to consider: these additions could not be made to the > KeyValueStore interface, since they'd be only applicable to > RocksDB-backed stores, but there is no RocksDBKeyValueStore, > which could be used to query the property. Rather, the rocks interface > is wrapped inside multiple layers that comprise the KeyValueStore. > Since these layers themselves are not public interfaces, it would be a > challenge to think of how to get access to these methods from > user-space code. > > On the other hand, could we just add new metrics to access those > properties? Offhand, it's unclear to me why you said they "may not be > exposed". > > Thanks, > -John > >> On Mon, Jul 15, 2019 at 5:11 AM Patrik Kleindl wrote: >> >> Hi >> Adding this here because I asked and may have found the answer: >> The memory consumption may not be exposed as RocksDB metrics, but they >> should be available as properties of the RocksDB instance itself. >> RocksDBStore could easily make this available by allowing access >> to db.getProperty. >> Available data (examples): >> rocksdb.block-cache-usage >> rocksdb.estimate-table-readers-mem >> rocksdb.cur-size-all-mem-tables >> >> Reference: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB >> and >> found the names of the properties here >> https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h >> >> Would it make sense to allow this kind of access? >> Should this be part of a follow-up KIP? >> >> best regards >> >> Patrik >> >>> On Wed, 19 Jun 2019 at 21:55, Bill Bejeck wrote: >>> >>> Hi Bruno, >>> >>> Just getting caught up on this KIP thread. Looks good to me, and I don't >>> have any additional comments to what's already been presented. >>> >>> Thanks, >>> Bill >>> On Wed, Jun 19, 2019 at 1:42 PM Bruno Cadonna wrote: John and Guozhang, thank you for your comments. @Guozhang could you please also vote on the voting thread so that we have all votes in one place. @John, the only situation I can think of where a non-uniform configuration of segments would make sense is to account for seasonality. But this would be a really advanced configuration IMO. Best, Bruno > On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > One last thought. I think it makes sense what you propose for merging > the metrics when a logical store is composed of multiple physical > stores. > > The basic standard for these metrics is that they should be relevant > to performance, and they should be controllable via configurations, > specifically via RocksDBConfigSetter. The config setter takes as input > the store name. For Key/Value stores, this name is visible in the tags > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > metric rocksdb-state-id=STORE_X is showing a low block cache hit > ratio, so I add a condition in my config setter to increase the block > cache size for stores named STORE_X. > > For this case where the logical store has multiple physical stores, > we're really talking about segmented stores, window stores and > segmented stores. In these stores, every segment has a different name > (logicalStoreName + "." + segmentId * segmentInterval), so the config > setter would need to use a prefix match with respect to the metric tag > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > doable. > > It's also perfectly reasonable, since segments get rotated out all the > time, it's implausible that you'd ever want a non-uniform > configuration over the segments in a store. For this reason, > specifically, it makes more sense just to summarize the metrics than > to present them individually. Might be worth documenting the > relationship, though. > > Thanks again, I'll vote now, > -John > > On Wed, Jun 19, 2019 at 12:03 PM John Roesler >>> wrote: >> >> Just taking a look over the metrics again, I had one thought... >> >> Stuff that happens in a background thread (like compaction metrics) >> can't directly identify compactions as a bottleneck from Streams' >> perspective. I.e., a DB might do a lot of compactions, but if those >> compactions never delay a write or read, then they cannot be a >> bottleneck. >> >> Thus, the "stall" metric should be the starting point for bottleneck >> identification, and then the flush/compaction metrics can be used to >> second
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hey Patrik, Since KIP-471 is already accepted, and since this idea is not a trivial extension of the KIP, I think we'd need to do a new KIP. Some points to consider: these additions could not be made to the KeyValueStore interface, since they'd be only applicable to RocksDB-backed stores, but there is no RocksDBKeyValueStore, which could be used to query the property. Rather, the rocks interface is wrapped inside multiple layers that comprise the KeyValueStore. Since these layers themselves are not public interfaces, it would be a challenge to think of how to get access to these methods from user-space code. On the other hand, could we just add new metrics to access those properties? Offhand, it's unclear to me why you said they "may not be exposed". Thanks, -John On Mon, Jul 15, 2019 at 5:11 AM Patrik Kleindl wrote: > > Hi > Adding this here because I asked and may have found the answer: > The memory consumption may not be exposed as RocksDB metrics, but they > should be available as properties of the RocksDB instance itself. > RocksDBStore could easily make this available by allowing access > to db.getProperty. > Available data (examples): > rocksdb.block-cache-usage > rocksdb.estimate-table-readers-mem > rocksdb.cur-size-all-mem-tables > > Reference: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB > and > found the names of the properties here > https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h > > Would it make sense to allow this kind of access? > Should this be part of a follow-up KIP? > > best regards > > Patrik > > On Wed, 19 Jun 2019 at 21:55, Bill Bejeck wrote: > > > Hi Bruno, > > > > Just getting caught up on this KIP thread. Looks good to me, and I don't > > have any additional comments to what's already been presented. > > > > Thanks, > > Bill > > > > On Wed, Jun 19, 2019 at 1:42 PM Bruno Cadonna wrote: > > > > > John and Guozhang, > > > > > > thank you for your comments. > > > > > > @Guozhang could you please also vote on the voting thread so that we > > > have all votes in one place. > > > > > > @John, the only situation I can think of where a non-uniform > > > configuration of segments would make sense is to account for > > > seasonality. But this would be a really advanced configuration IMO. > > > > > > Best, > > > Bruno > > > > > > On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > > > > > > > One last thought. I think it makes sense what you propose for merging > > > > the metrics when a logical store is composed of multiple physical > > > > stores. > > > > > > > > The basic standard for these metrics is that they should be relevant > > > > to performance, and they should be controllable via configurations, > > > > specifically via RocksDBConfigSetter. The config setter takes as input > > > > the store name. For Key/Value stores, this name is visible in the tags > > > > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > > > > metric rocksdb-state-id=STORE_X is showing a low block cache hit > > > > ratio, so I add a condition in my config setter to increase the block > > > > cache size for stores named STORE_X. > > > > > > > > For this case where the logical store has multiple physical stores, > > > > we're really talking about segmented stores, window stores and > > > > segmented stores. In these stores, every segment has a different name > > > > (logicalStoreName + "." + segmentId * segmentInterval), so the config > > > > setter would need to use a prefix match with respect to the metric tag > > > > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > > > > doable. > > > > > > > > It's also perfectly reasonable, since segments get rotated out all the > > > > time, it's implausible that you'd ever want a non-uniform > > > > configuration over the segments in a store. For this reason, > > > > specifically, it makes more sense just to summarize the metrics than > > > > to present them individually. Might be worth documenting the > > > > relationship, though. > > > > > > > > Thanks again, I'll vote now, > > > > -John > > > > > > > > On Wed, Jun 19, 2019 at 12:03 PM John Roesler > > wrote: > > > > > > > > > > Just taking a look over the metrics again, I had one thought... > > > > > > > > > > Stuff that happens in a background thread (like compaction metrics) > > > > > can't directly identify compactions as a bottleneck from Streams' > > > > > perspective. I.e., a DB might do a lot of compactions, but if those > > > > > compactions never delay a write or read, then they cannot be a > > > > > bottleneck. > > > > > > > > > > Thus, the "stall" metric should be the starting point for bottleneck > > > > > identification, and then the flush/compaction metrics can be used to > > > > > secondarily identify what to do to relieve the bottleneck. > > > > > > > > > > This doesn't affect the metrics you proposed, but I'd suggest saying > > > > > something to this effect in whatever documentation or descriptions we >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Adding this here because I asked and may have found the answer: The memory consumption may not be exposed as RocksDB metrics, but they should be available as properties of the RocksDB instance itself. RocksDBStore could easily make this available by allowing access to db.getProperty. Available data (examples): rocksdb.block-cache-usage rocksdb.estimate-table-readers-mem rocksdb.cur-size-all-mem-tables Reference: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB and found the names of the properties here https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h Would it make sense to allow this kind of access? Should this be part of a follow-up KIP? best regards Patrik On Wed, 19 Jun 2019 at 21:55, Bill Bejeck wrote: > Hi Bruno, > > Just getting caught up on this KIP thread. Looks good to me, and I don't > have any additional comments to what's already been presented. > > Thanks, > Bill > > On Wed, Jun 19, 2019 at 1:42 PM Bruno Cadonna wrote: > > > John and Guozhang, > > > > thank you for your comments. > > > > @Guozhang could you please also vote on the voting thread so that we > > have all votes in one place. > > > > @John, the only situation I can think of where a non-uniform > > configuration of segments would make sense is to account for > > seasonality. But this would be a really advanced configuration IMO. > > > > Best, > > Bruno > > > > On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > > > > > One last thought. I think it makes sense what you propose for merging > > > the metrics when a logical store is composed of multiple physical > > > stores. > > > > > > The basic standard for these metrics is that they should be relevant > > > to performance, and they should be controllable via configurations, > > > specifically via RocksDBConfigSetter. The config setter takes as input > > > the store name. For Key/Value stores, this name is visible in the tags > > > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > > > metric rocksdb-state-id=STORE_X is showing a low block cache hit > > > ratio, so I add a condition in my config setter to increase the block > > > cache size for stores named STORE_X. > > > > > > For this case where the logical store has multiple physical stores, > > > we're really talking about segmented stores, window stores and > > > segmented stores. In these stores, every segment has a different name > > > (logicalStoreName + "." + segmentId * segmentInterval), so the config > > > setter would need to use a prefix match with respect to the metric tag > > > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > > > doable. > > > > > > It's also perfectly reasonable, since segments get rotated out all the > > > time, it's implausible that you'd ever want a non-uniform > > > configuration over the segments in a store. For this reason, > > > specifically, it makes more sense just to summarize the metrics than > > > to present them individually. Might be worth documenting the > > > relationship, though. > > > > > > Thanks again, I'll vote now, > > > -John > > > > > > On Wed, Jun 19, 2019 at 12:03 PM John Roesler > wrote: > > > > > > > > Just taking a look over the metrics again, I had one thought... > > > > > > > > Stuff that happens in a background thread (like compaction metrics) > > > > can't directly identify compactions as a bottleneck from Streams' > > > > perspective. I.e., a DB might do a lot of compactions, but if those > > > > compactions never delay a write or read, then they cannot be a > > > > bottleneck. > > > > > > > > Thus, the "stall" metric should be the starting point for bottleneck > > > > identification, and then the flush/compaction metrics can be used to > > > > secondarily identify what to do to relieve the bottleneck. > > > > > > > > This doesn't affect the metrics you proposed, but I'd suggest saying > > > > something to this effect in whatever documentation or descriptions we > > > > provide. > > > > > > > > Thanks, > > > > -John > > > > > > > > On Wed, Jun 19, 2019 at 11:25 AM John Roesler > > wrote: > > > > > > > > > > Thanks for the updates. > > > > > > > > > > Personally, I'd be in favor of not going out on a limb with > > > > > unsupported metrics APIs. We should take care to make sure that > what > > > > > we add in KIP-471 is stable and well supported, even if it's not > the > > > > > complete picture. We can always do follow-on work to tackle complex > > > > > metrics as an isolated design exercise. > > > > > > > > > > Just my two cents. > > > > > Thanks, > > > > > -John > > > > > > > > > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna > > wrote: > > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > Regarding your comments about the wiki page: > > > > > > > > > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > > > > > > > > > 2) Yes, I used the wrong term. All hit related metrics are > ratios. > > I > > > > > > corrected the names of the affected
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Bruno, Just getting caught up on this KIP thread. Looks good to me, and I don't have any additional comments to what's already been presented. Thanks, Bill On Wed, Jun 19, 2019 at 1:42 PM Bruno Cadonna wrote: > John and Guozhang, > > thank you for your comments. > > @Guozhang could you please also vote on the voting thread so that we > have all votes in one place. > > @John, the only situation I can think of where a non-uniform > configuration of segments would make sense is to account for > seasonality. But this would be a really advanced configuration IMO. > > Best, > Bruno > > On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > > > One last thought. I think it makes sense what you propose for merging > > the metrics when a logical store is composed of multiple physical > > stores. > > > > The basic standard for these metrics is that they should be relevant > > to performance, and they should be controllable via configurations, > > specifically via RocksDBConfigSetter. The config setter takes as input > > the store name. For Key/Value stores, this name is visible in the tags > > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > > metric rocksdb-state-id=STORE_X is showing a low block cache hit > > ratio, so I add a condition in my config setter to increase the block > > cache size for stores named STORE_X. > > > > For this case where the logical store has multiple physical stores, > > we're really talking about segmented stores, window stores and > > segmented stores. In these stores, every segment has a different name > > (logicalStoreName + "." + segmentId * segmentInterval), so the config > > setter would need to use a prefix match with respect to the metric tag > > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > > doable. > > > > It's also perfectly reasonable, since segments get rotated out all the > > time, it's implausible that you'd ever want a non-uniform > > configuration over the segments in a store. For this reason, > > specifically, it makes more sense just to summarize the metrics than > > to present them individually. Might be worth documenting the > > relationship, though. > > > > Thanks again, I'll vote now, > > -John > > > > On Wed, Jun 19, 2019 at 12:03 PM John Roesler wrote: > > > > > > Just taking a look over the metrics again, I had one thought... > > > > > > Stuff that happens in a background thread (like compaction metrics) > > > can't directly identify compactions as a bottleneck from Streams' > > > perspective. I.e., a DB might do a lot of compactions, but if those > > > compactions never delay a write or read, then they cannot be a > > > bottleneck. > > > > > > Thus, the "stall" metric should be the starting point for bottleneck > > > identification, and then the flush/compaction metrics can be used to > > > secondarily identify what to do to relieve the bottleneck. > > > > > > This doesn't affect the metrics you proposed, but I'd suggest saying > > > something to this effect in whatever documentation or descriptions we > > > provide. > > > > > > Thanks, > > > -John > > > > > > On Wed, Jun 19, 2019 at 11:25 AM John Roesler > wrote: > > > > > > > > Thanks for the updates. > > > > > > > > Personally, I'd be in favor of not going out on a limb with > > > > unsupported metrics APIs. We should take care to make sure that what > > > > we add in KIP-471 is stable and well supported, even if it's not the > > > > complete picture. We can always do follow-on work to tackle complex > > > > metrics as an isolated design exercise. > > > > > > > > Just my two cents. > > > > Thanks, > > > > -John > > > > > > > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna > wrote: > > > > > > > > > > Hi Guozhang, > > > > > > > > > > Regarding your comments about the wiki page: > > > > > > > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > > > > > > > 2) Yes, I used the wrong term. All hit related metrics are ratios. > I > > > > > corrected the names of the affected metrics. > > > > > > > > > > Regarding your meta comments: > > > > > > > > > > 1) The plan is to expose the hit ratio. I used the wrong term. The > > > > > formulas compute ratios. Regarding your question about a metric to > > > > > know from where a read is served when it is not in the memtable, > there > > > > > are metrics in RocksDB that give you the number of get() queries > that > > > > > are served from L0, L1, and L2_AND_UP. I could not find any metric > > > > > that give you information about whether a query was served from > disk > > > > > vs. OS cache. One metric that could be used to indirectly measure > > > > > whether disk or OS cache is accessed seems to be > READ_BLOCK_GET_MICROS > > > > > that gives you the time for an IO read of a block. If it is high, > it > > > > > was read from disk, otherwise from the OS cache. A similar > strategy to > > > > > monitor the performance is described in [1]. DISCLAIMER: > > > > > READ_BLOCK_GET_MICROS is not d
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
John and Guozhang, thank you for your comments. @Guozhang could you please also vote on the voting thread so that we have all votes in one place. @John, the only situation I can think of where a non-uniform configuration of segments would make sense is to account for seasonality. But this would be a really advanced configuration IMO. Best, Bruno On Wed, Jun 19, 2019 at 7:18 PM John Roesler wrote: > > One last thought. I think it makes sense what you propose for merging > the metrics when a logical store is composed of multiple physical > stores. > > The basic standard for these metrics is that they should be relevant > to performance, and they should be controllable via configurations, > specifically via RocksDBConfigSetter. The config setter takes as input > the store name. For Key/Value stores, this name is visible in the tags > as "rocksdb-state-id", so the workflow is that I notice (e.g.) the > metric rocksdb-state-id=STORE_X is showing a low block cache hit > ratio, so I add a condition in my config setter to increase the block > cache size for stores named STORE_X. > > For this case where the logical store has multiple physical stores, > we're really talking about segmented stores, window stores and > segmented stores. In these stores, every segment has a different name > (logicalStoreName + "." + segmentId * segmentInterval), so the config > setter would need to use a prefix match with respect to the metric tag > (e.g.) rocksdb-window-state-id. But at the same time, this is totally > doable. > > It's also perfectly reasonable, since segments get rotated out all the > time, it's implausible that you'd ever want a non-uniform > configuration over the segments in a store. For this reason, > specifically, it makes more sense just to summarize the metrics than > to present them individually. Might be worth documenting the > relationship, though. > > Thanks again, I'll vote now, > -John > > On Wed, Jun 19, 2019 at 12:03 PM John Roesler wrote: > > > > Just taking a look over the metrics again, I had one thought... > > > > Stuff that happens in a background thread (like compaction metrics) > > can't directly identify compactions as a bottleneck from Streams' > > perspective. I.e., a DB might do a lot of compactions, but if those > > compactions never delay a write or read, then they cannot be a > > bottleneck. > > > > Thus, the "stall" metric should be the starting point for bottleneck > > identification, and then the flush/compaction metrics can be used to > > secondarily identify what to do to relieve the bottleneck. > > > > This doesn't affect the metrics you proposed, but I'd suggest saying > > something to this effect in whatever documentation or descriptions we > > provide. > > > > Thanks, > > -John > > > > On Wed, Jun 19, 2019 at 11:25 AM John Roesler wrote: > > > > > > Thanks for the updates. > > > > > > Personally, I'd be in favor of not going out on a limb with > > > unsupported metrics APIs. We should take care to make sure that what > > > we add in KIP-471 is stable and well supported, even if it's not the > > > complete picture. We can always do follow-on work to tackle complex > > > metrics as an isolated design exercise. > > > > > > Just my two cents. > > > Thanks, > > > -John > > > > > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna wrote: > > > > > > > > Hi Guozhang, > > > > > > > > Regarding your comments about the wiki page: > > > > > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > > > > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > > > > corrected the names of the affected metrics. > > > > > > > > Regarding your meta comments: > > > > > > > > 1) The plan is to expose the hit ratio. I used the wrong term. The > > > > formulas compute ratios. Regarding your question about a metric to > > > > know from where a read is served when it is not in the memtable, there > > > > are metrics in RocksDB that give you the number of get() queries that > > > > are served from L0, L1, and L2_AND_UP. I could not find any metric > > > > that give you information about whether a query was served from disk > > > > vs. OS cache. One metric that could be used to indirectly measure > > > > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > > > > that gives you the time for an IO read of a block. If it is high, it > > > > was read from disk, otherwise from the OS cache. A similar strategy to > > > > monitor the performance is described in [1]. DISCLAIMER: > > > > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > > > > code to understand its meaning. I could have missed something. > > > > > > > > 2) There are some additional compaction statistics that contain sizes > > > > of files on disk and numbers about write amplification that you can > > > > get programmatically in RocksDB, but they are for debugging purposes > > > > [2]. To get this data and publish it into a metric, one has to parse a > > > > string. Si
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
One last thought. I think it makes sense what you propose for merging the metrics when a logical store is composed of multiple physical stores. The basic standard for these metrics is that they should be relevant to performance, and they should be controllable via configurations, specifically via RocksDBConfigSetter. The config setter takes as input the store name. For Key/Value stores, this name is visible in the tags as "rocksdb-state-id", so the workflow is that I notice (e.g.) the metric rocksdb-state-id=STORE_X is showing a low block cache hit ratio, so I add a condition in my config setter to increase the block cache size for stores named STORE_X. For this case where the logical store has multiple physical stores, we're really talking about segmented stores, window stores and segmented stores. In these stores, every segment has a different name (logicalStoreName + "." + segmentId * segmentInterval), so the config setter would need to use a prefix match with respect to the metric tag (e.g.) rocksdb-window-state-id. But at the same time, this is totally doable. It's also perfectly reasonable, since segments get rotated out all the time, it's implausible that you'd ever want a non-uniform configuration over the segments in a store. For this reason, specifically, it makes more sense just to summarize the metrics than to present them individually. Might be worth documenting the relationship, though. Thanks again, I'll vote now, -John On Wed, Jun 19, 2019 at 12:03 PM John Roesler wrote: > > Just taking a look over the metrics again, I had one thought... > > Stuff that happens in a background thread (like compaction metrics) > can't directly identify compactions as a bottleneck from Streams' > perspective. I.e., a DB might do a lot of compactions, but if those > compactions never delay a write or read, then they cannot be a > bottleneck. > > Thus, the "stall" metric should be the starting point for bottleneck > identification, and then the flush/compaction metrics can be used to > secondarily identify what to do to relieve the bottleneck. > > This doesn't affect the metrics you proposed, but I'd suggest saying > something to this effect in whatever documentation or descriptions we > provide. > > Thanks, > -John > > On Wed, Jun 19, 2019 at 11:25 AM John Roesler wrote: > > > > Thanks for the updates. > > > > Personally, I'd be in favor of not going out on a limb with > > unsupported metrics APIs. We should take care to make sure that what > > we add in KIP-471 is stable and well supported, even if it's not the > > complete picture. We can always do follow-on work to tackle complex > > metrics as an isolated design exercise. > > > > Just my two cents. > > Thanks, > > -John > > > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna wrote: > > > > > > Hi Guozhang, > > > > > > Regarding your comments about the wiki page: > > > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > > > corrected the names of the affected metrics. > > > > > > Regarding your meta comments: > > > > > > 1) The plan is to expose the hit ratio. I used the wrong term. The > > > formulas compute ratios. Regarding your question about a metric to > > > know from where a read is served when it is not in the memtable, there > > > are metrics in RocksDB that give you the number of get() queries that > > > are served from L0, L1, and L2_AND_UP. I could not find any metric > > > that give you information about whether a query was served from disk > > > vs. OS cache. One metric that could be used to indirectly measure > > > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > > > that gives you the time for an IO read of a block. If it is high, it > > > was read from disk, otherwise from the OS cache. A similar strategy to > > > monitor the performance is described in [1]. DISCLAIMER: > > > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > > > code to understand its meaning. I could have missed something. > > > > > > 2) There are some additional compaction statistics that contain sizes > > > of files on disk and numbers about write amplification that you can > > > get programmatically in RocksDB, but they are for debugging purposes > > > [2]. To get this data and publish it into a metric, one has to parse a > > > string. Since this data is for debugging purposes, I do not know how > > > stable the output format is. One thing, we could do, is to dump the > > > string with the compaction statistics into our log files at DEBUG > > > level. But that is outside of the scope of this KIP. > > > > > > Best, > > > Bruno > > > > > > [1] > > > https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency > > > [2] > > > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics > > > > > > On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang wrote: >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Bruno, thanks for the clarification. I agree that we should not rely on parsing strings to expose as metrics since they are 1) not very reliable and also 2) may evolve its format / representation over time. I think we can potentially add some documentations aligned with your explanations above to educate users how to predicate their capacity on memory and disks, rather than exposing the metrics; these would be not be required in this KIP then. Maybe we can create a JIRA ticket for adding web docs to educate users investigating their rocksDB behaviors, also including John's comments above? As for the KIP itself, I'm +1. Guozhang On Wed, Jun 19, 2019 at 10:04 AM John Roesler wrote: > Just taking a look over the metrics again, I had one thought... > > Stuff that happens in a background thread (like compaction metrics) > can't directly identify compactions as a bottleneck from Streams' > perspective. I.e., a DB might do a lot of compactions, but if those > compactions never delay a write or read, then they cannot be a > bottleneck. > > Thus, the "stall" metric should be the starting point for bottleneck > identification, and then the flush/compaction metrics can be used to > secondarily identify what to do to relieve the bottleneck. > > This doesn't affect the metrics you proposed, but I'd suggest saying > something to this effect in whatever documentation or descriptions we > provide. > > Thanks, > -John > > On Wed, Jun 19, 2019 at 11:25 AM John Roesler wrote: > > > > Thanks for the updates. > > > > Personally, I'd be in favor of not going out on a limb with > > unsupported metrics APIs. We should take care to make sure that what > > we add in KIP-471 is stable and well supported, even if it's not the > > complete picture. We can always do follow-on work to tackle complex > > metrics as an isolated design exercise. > > > > Just my two cents. > > Thanks, > > -John > > > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna > wrote: > > > > > > Hi Guozhang, > > > > > > Regarding your comments about the wiki page: > > > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > > > corrected the names of the affected metrics. > > > > > > Regarding your meta comments: > > > > > > 1) The plan is to expose the hit ratio. I used the wrong term. The > > > formulas compute ratios. Regarding your question about a metric to > > > know from where a read is served when it is not in the memtable, there > > > are metrics in RocksDB that give you the number of get() queries that > > > are served from L0, L1, and L2_AND_UP. I could not find any metric > > > that give you information about whether a query was served from disk > > > vs. OS cache. One metric that could be used to indirectly measure > > > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > > > that gives you the time for an IO read of a block. If it is high, it > > > was read from disk, otherwise from the OS cache. A similar strategy to > > > monitor the performance is described in [1]. DISCLAIMER: > > > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > > > code to understand its meaning. I could have missed something. > > > > > > 2) There are some additional compaction statistics that contain sizes > > > of files on disk and numbers about write amplification that you can > > > get programmatically in RocksDB, but they are for debugging purposes > > > [2]. To get this data and publish it into a metric, one has to parse a > > > string. Since this data is for debugging purposes, I do not know how > > > stable the output format is. One thing, we could do, is to dump the > > > string with the compaction statistics into our log files at DEBUG > > > level. But that is outside of the scope of this KIP. > > > > > > Best, > > > Bruno > > > > > > [1] > https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency > > > [2] > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics > > > > > > On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang > wrote: > > > > > > > > Hello Bruno, > > > > > > > > I've read through the aggregation section and I think they look good > to me. > > > > There are a few minor comments about the wiki page itself: > > > > > > > > 1) A state store might consist of multiple state stores -> You mean a > > > > `logical` state store be consistent of multiple `physical` store > instances? > > > > > > > > 2) The "Hit Rates" calculation seems to be referring to the `Hit > Ratio` > > > > (which is a percentage) than `Hit Rate`? > > > > > > > > And a couple further meta comments: > > > > > > > > 1) For memtable / block cache, instead of the hit-rate do you think > we > > > > should expose the hit-ratio? I felt it is more useful for users to > debug > > > > what's the root cause of unexpected slow performance. > > > > > > > > And for block cache misses, i
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Just taking a look over the metrics again, I had one thought... Stuff that happens in a background thread (like compaction metrics) can't directly identify compactions as a bottleneck from Streams' perspective. I.e., a DB might do a lot of compactions, but if those compactions never delay a write or read, then they cannot be a bottleneck. Thus, the "stall" metric should be the starting point for bottleneck identification, and then the flush/compaction metrics can be used to secondarily identify what to do to relieve the bottleneck. This doesn't affect the metrics you proposed, but I'd suggest saying something to this effect in whatever documentation or descriptions we provide. Thanks, -John On Wed, Jun 19, 2019 at 11:25 AM John Roesler wrote: > > Thanks for the updates. > > Personally, I'd be in favor of not going out on a limb with > unsupported metrics APIs. We should take care to make sure that what > we add in KIP-471 is stable and well supported, even if it's not the > complete picture. We can always do follow-on work to tackle complex > metrics as an isolated design exercise. > > Just my two cents. > Thanks, > -John > > On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna wrote: > > > > Hi Guozhang, > > > > Regarding your comments about the wiki page: > > > > 1) Exactly, I rephrased the paragraph to make it more clear. > > > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > > corrected the names of the affected metrics. > > > > Regarding your meta comments: > > > > 1) The plan is to expose the hit ratio. I used the wrong term. The > > formulas compute ratios. Regarding your question about a metric to > > know from where a read is served when it is not in the memtable, there > > are metrics in RocksDB that give you the number of get() queries that > > are served from L0, L1, and L2_AND_UP. I could not find any metric > > that give you information about whether a query was served from disk > > vs. OS cache. One metric that could be used to indirectly measure > > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > > that gives you the time for an IO read of a block. If it is high, it > > was read from disk, otherwise from the OS cache. A similar strategy to > > monitor the performance is described in [1]. DISCLAIMER: > > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > > code to understand its meaning. I could have missed something. > > > > 2) There are some additional compaction statistics that contain sizes > > of files on disk and numbers about write amplification that you can > > get programmatically in RocksDB, but they are for debugging purposes > > [2]. To get this data and publish it into a metric, one has to parse a > > string. Since this data is for debugging purposes, I do not know how > > stable the output format is. One thing, we could do, is to dump the > > string with the compaction statistics into our log files at DEBUG > > level. But that is outside of the scope of this KIP. > > > > Best, > > Bruno > > > > [1] > > https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency > > [2] > > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics > > > > On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang wrote: > > > > > > Hello Bruno, > > > > > > I've read through the aggregation section and I think they look good to > > > me. > > > There are a few minor comments about the wiki page itself: > > > > > > 1) A state store might consist of multiple state stores -> You mean a > > > `logical` state store be consistent of multiple `physical` store > > > instances? > > > > > > 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` > > > (which is a percentage) than `Hit Rate`? > > > > > > And a couple further meta comments: > > > > > > 1) For memtable / block cache, instead of the hit-rate do you think we > > > should expose the hit-ratio? I felt it is more useful for users to debug > > > what's the root cause of unexpected slow performance. > > > > > > And for block cache misses, is it easy to provide a metric as of "target > > > read" of where a read is served (from which level, either in OS cache or > > > in > > > SST files), similar to Fig.11 in > > > http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? > > > > > > 2) As @Patrik mentioned, is there a good way we can expose the total > > > amount > > > of memory and disk usage for each state store as well? I think it would > > > also be very helpful for users to understand their capacity needs and read > > > / write amplifications. > > > > > > > > > Guozhang > > > > > > On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna wrote: > > > > > > > Hi, > > > > > > > > I decided to go for the option in which metrics are exposed for each > > > > logical state store. I revisited the KIP correspondingly and added a > > > > section on how to aggregate metrics over multiple physical RocksDB > > > > in
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Thanks for the updates. Personally, I'd be in favor of not going out on a limb with unsupported metrics APIs. We should take care to make sure that what we add in KIP-471 is stable and well supported, even if it's not the complete picture. We can always do follow-on work to tackle complex metrics as an isolated design exercise. Just my two cents. Thanks, -John On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna wrote: > > Hi Guozhang, > > Regarding your comments about the wiki page: > > 1) Exactly, I rephrased the paragraph to make it more clear. > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > corrected the names of the affected metrics. > > Regarding your meta comments: > > 1) The plan is to expose the hit ratio. I used the wrong term. The > formulas compute ratios. Regarding your question about a metric to > know from where a read is served when it is not in the memtable, there > are metrics in RocksDB that give you the number of get() queries that > are served from L0, L1, and L2_AND_UP. I could not find any metric > that give you information about whether a query was served from disk > vs. OS cache. One metric that could be used to indirectly measure > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > that gives you the time for an IO read of a block. If it is high, it > was read from disk, otherwise from the OS cache. A similar strategy to > monitor the performance is described in [1]. DISCLAIMER: > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > code to understand its meaning. I could have missed something. > > 2) There are some additional compaction statistics that contain sizes > of files on disk and numbers about write amplification that you can > get programmatically in RocksDB, but they are for debugging purposes > [2]. To get this data and publish it into a metric, one has to parse a > string. Since this data is for debugging purposes, I do not know how > stable the output format is. One thing, we could do, is to dump the > string with the compaction statistics into our log files at DEBUG > level. But that is outside of the scope of this KIP. > > Best, > Bruno > > [1] > https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency > [2] > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics > > On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang wrote: > > > > Hello Bruno, > > > > I've read through the aggregation section and I think they look good to me. > > There are a few minor comments about the wiki page itself: > > > > 1) A state store might consist of multiple state stores -> You mean a > > `logical` state store be consistent of multiple `physical` store instances? > > > > 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` > > (which is a percentage) than `Hit Rate`? > > > > And a couple further meta comments: > > > > 1) For memtable / block cache, instead of the hit-rate do you think we > > should expose the hit-ratio? I felt it is more useful for users to debug > > what's the root cause of unexpected slow performance. > > > > And for block cache misses, is it easy to provide a metric as of "target > > read" of where a read is served (from which level, either in OS cache or in > > SST files), similar to Fig.11 in > > http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? > > > > 2) As @Patrik mentioned, is there a good way we can expose the total amount > > of memory and disk usage for each state store as well? I think it would > > also be very helpful for users to understand their capacity needs and read > > / write amplifications. > > > > > > Guozhang > > > > On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna wrote: > > > > > Hi, > > > > > > I decided to go for the option in which metrics are exposed for each > > > logical state store. I revisited the KIP correspondingly and added a > > > section on how to aggregate metrics over multiple physical RocksDB > > > instances within one logical state store. Would be great, if you could > > > take a look and give feedback. If nobody has complaints about the > > > chosen option I would proceed with voting on this KIP since this was > > > the last open question. > > > > > > Best, > > > Bruno > > > > > > On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl wrote: > > > > > > > > Hi Sophie > > > > This will be a good change, I have been thinking about proposing > > > something similar or even passing the properties per store. > > > > RocksDB should probably know how much memory was reserved but maybe does > > > not expose it. > > > > We are limiting it already as you suggested but this is a rather crude > > > tool. > > > > Especially in a larger topology with mixed loads par topic it would be > > > helpful to get more insights which store puts a lot of load on memory. > > > > Regarding the limiting capability, I think I remember reading that those > > > only affect some parts of the memor
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Guozhang, Regarding your comments about the wiki page: 1) Exactly, I rephrased the paragraph to make it more clear. 2) Yes, I used the wrong term. All hit related metrics are ratios. I corrected the names of the affected metrics. Regarding your meta comments: 1) The plan is to expose the hit ratio. I used the wrong term. The formulas compute ratios. Regarding your question about a metric to know from where a read is served when it is not in the memtable, there are metrics in RocksDB that give you the number of get() queries that are served from L0, L1, and L2_AND_UP. I could not find any metric that give you information about whether a query was served from disk vs. OS cache. One metric that could be used to indirectly measure whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS that gives you the time for an IO read of a block. If it is high, it was read from disk, otherwise from the OS cache. A similar strategy to monitor the performance is described in [1]. DISCLAIMER: READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ code to understand its meaning. I could have missed something. 2) There are some additional compaction statistics that contain sizes of files on disk and numbers about write amplification that you can get programmatically in RocksDB, but they are for debugging purposes [2]. To get this data and publish it into a metric, one has to parse a string. Since this data is for debugging purposes, I do not know how stable the output format is. One thing, we could do, is to dump the string with the compaction statistics into our log files at DEBUG level. But that is outside of the scope of this KIP. Best, Bruno [1] https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency [2] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang wrote: > > Hello Bruno, > > I've read through the aggregation section and I think they look good to me. > There are a few minor comments about the wiki page itself: > > 1) A state store might consist of multiple state stores -> You mean a > `logical` state store be consistent of multiple `physical` store instances? > > 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` > (which is a percentage) than `Hit Rate`? > > And a couple further meta comments: > > 1) For memtable / block cache, instead of the hit-rate do you think we > should expose the hit-ratio? I felt it is more useful for users to debug > what's the root cause of unexpected slow performance. > > And for block cache misses, is it easy to provide a metric as of "target > read" of where a read is served (from which level, either in OS cache or in > SST files), similar to Fig.11 in > http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? > > 2) As @Patrik mentioned, is there a good way we can expose the total amount > of memory and disk usage for each state store as well? I think it would > also be very helpful for users to understand their capacity needs and read > / write amplifications. > > > Guozhang > > On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna wrote: > > > Hi, > > > > I decided to go for the option in which metrics are exposed for each > > logical state store. I revisited the KIP correspondingly and added a > > section on how to aggregate metrics over multiple physical RocksDB > > instances within one logical state store. Would be great, if you could > > take a look and give feedback. If nobody has complaints about the > > chosen option I would proceed with voting on this KIP since this was > > the last open question. > > > > Best, > > Bruno > > > > On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl wrote: > > > > > > Hi Sophie > > > This will be a good change, I have been thinking about proposing > > something similar or even passing the properties per store. > > > RocksDB should probably know how much memory was reserved but maybe does > > not expose it. > > > We are limiting it already as you suggested but this is a rather crude > > tool. > > > Especially in a larger topology with mixed loads par topic it would be > > helpful to get more insights which store puts a lot of load on memory. > > > Regarding the limiting capability, I think I remember reading that those > > only affect some parts of the memory and others can still exceed this > > limit. I‘ll try to look up the difference. > > > Best regards > > > Patrik > > > > > > > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman < > > sop...@confluent.io>: > > > > > > > > Hi Patrik, > > > > > > > > As of 2.3 you will be able to use the RocksDBConfigSetter to > > effectively > > > > bound the total memory used by RocksDB for a single app instance. You > > > > should already be able to limit the memory used per rocksdb store, > > though > > > > as you mention there can be a lot of them. I'm not sure you can > > monitor the > > > > memory usage if you are not li
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hello Bruno, I've read through the aggregation section and I think they look good to me. There are a few minor comments about the wiki page itself: 1) A state store might consist of multiple state stores -> You mean a `logical` state store be consistent of multiple `physical` store instances? 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` (which is a percentage) than `Hit Rate`? And a couple further meta comments: 1) For memtable / block cache, instead of the hit-rate do you think we should expose the hit-ratio? I felt it is more useful for users to debug what's the root cause of unexpected slow performance. And for block cache misses, is it easy to provide a metric as of "target read" of where a read is served (from which level, either in OS cache or in SST files), similar to Fig.11 in http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? 2) As @Patrik mentioned, is there a good way we can expose the total amount of memory and disk usage for each state store as well? I think it would also be very helpful for users to understand their capacity needs and read / write amplifications. Guozhang On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna wrote: > Hi, > > I decided to go for the option in which metrics are exposed for each > logical state store. I revisited the KIP correspondingly and added a > section on how to aggregate metrics over multiple physical RocksDB > instances within one logical state store. Would be great, if you could > take a look and give feedback. If nobody has complaints about the > chosen option I would proceed with voting on this KIP since this was > the last open question. > > Best, > Bruno > > On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl wrote: > > > > Hi Sophie > > This will be a good change, I have been thinking about proposing > something similar or even passing the properties per store. > > RocksDB should probably know how much memory was reserved but maybe does > not expose it. > > We are limiting it already as you suggested but this is a rather crude > tool. > > Especially in a larger topology with mixed loads par topic it would be > helpful to get more insights which store puts a lot of load on memory. > > Regarding the limiting capability, I think I remember reading that those > only affect some parts of the memory and others can still exceed this > limit. I‘ll try to look up the difference. > > Best regards > > Patrik > > > > > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman < > sop...@confluent.io>: > > > > > > Hi Patrik, > > > > > > As of 2.3 you will be able to use the RocksDBConfigSetter to > effectively > > > bound the total memory used by RocksDB for a single app instance. You > > > should already be able to limit the memory used per rocksdb store, > though > > > as you mention there can be a lot of them. I'm not sure you can > monitor the > > > memory usage if you are not limiting it though. > > > > > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl > wrote: > > >> > > >> Hi > > >> Thanks Bruno for the KIP, this is a very good idea. > > >> > > >> I have one question, are there metrics available for the memory > consumption > > >> of RocksDB? > > >> As they are running outside the JVM we have run into issues because > they > > >> were using all the other memory. > > >> And with multiple streams applications on the same machine, each with > > >> several KTables and 10+ partitions per topic the number of stores can > get > > >> out of hand pretty easily. > > >> Or did I miss something obvious how those can be monitored better? > > >> > > >> best regards > > >> > > >> Patrik > > >> > > >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna > wrote: > > >>> > > >>> Hi all, > > >>> > > >>> this KIP describes the extension of the Kafka Streams' metrics to > include > > >>> RocksDB's internal statistics. > > >>> > > >>> Please have a look at it and let me know what you think. Since I am > not a > > >>> RocksDB expert, I am thankful for any additional pair of eyes that > > >>> evaluates this KIP. > > >>> > > >>> > > >>> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > >>> > > >>> Best regards, > > >>> Bruno > > >>> > > >> > -- -- Guozhang
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi, I decided to go for the option in which metrics are exposed for each logical state store. I revisited the KIP correspondingly and added a section on how to aggregate metrics over multiple physical RocksDB instances within one logical state store. Would be great, if you could take a look and give feedback. If nobody has complaints about the chosen option I would proceed with voting on this KIP since this was the last open question. Best, Bruno On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl wrote: > > Hi Sophie > This will be a good change, I have been thinking about proposing something > similar or even passing the properties per store. > RocksDB should probably know how much memory was reserved but maybe does not > expose it. > We are limiting it already as you suggested but this is a rather crude tool. > Especially in a larger topology with mixed loads par topic it would be > helpful to get more insights which store puts a lot of load on memory. > Regarding the limiting capability, I think I remember reading that those only > affect some parts of the memory and others can still exceed this limit. I‘ll > try to look up the difference. > Best regards > Patrik > > > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman : > > > > Hi Patrik, > > > > As of 2.3 you will be able to use the RocksDBConfigSetter to effectively > > bound the total memory used by RocksDB for a single app instance. You > > should already be able to limit the memory used per rocksdb store, though > > as you mention there can be a lot of them. I'm not sure you can monitor the > > memory usage if you are not limiting it though. > > > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl wrote: > >> > >> Hi > >> Thanks Bruno for the KIP, this is a very good idea. > >> > >> I have one question, are there metrics available for the memory consumption > >> of RocksDB? > >> As they are running outside the JVM we have run into issues because they > >> were using all the other memory. > >> And with multiple streams applications on the same machine, each with > >> several KTables and 10+ partitions per topic the number of stores can get > >> out of hand pretty easily. > >> Or did I miss something obvious how those can be monitored better? > >> > >> best regards > >> > >> Patrik > >> > >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna wrote: > >>> > >>> Hi all, > >>> > >>> this KIP describes the extension of the Kafka Streams' metrics to include > >>> RocksDB's internal statistics. > >>> > >>> Please have a look at it and let me know what you think. Since I am not a > >>> RocksDB expert, I am thankful for any additional pair of eyes that > >>> evaluates this KIP. > >>> > >>> > >>> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > >>> > >>> Best regards, > >>> Bruno > >>> > >>
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Sophie This will be a good change, I have been thinking about proposing something similar or even passing the properties per store. RocksDB should probably know how much memory was reserved but maybe does not expose it. We are limiting it already as you suggested but this is a rather crude tool. Especially in a larger topology with mixed loads par topic it would be helpful to get more insights which store puts a lot of load on memory. Regarding the limiting capability, I think I remember reading that those only affect some parts of the memory and others can still exceed this limit. I‘ll try to look up the difference. Best regards Patrik > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman : > > Hi Patrik, > > As of 2.3 you will be able to use the RocksDBConfigSetter to effectively > bound the total memory used by RocksDB for a single app instance. You > should already be able to limit the memory used per rocksdb store, though > as you mention there can be a lot of them. I'm not sure you can monitor the > memory usage if you are not limiting it though. > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl wrote: >> >> Hi >> Thanks Bruno for the KIP, this is a very good idea. >> >> I have one question, are there metrics available for the memory consumption >> of RocksDB? >> As they are running outside the JVM we have run into issues because they >> were using all the other memory. >> And with multiple streams applications on the same machine, each with >> several KTables and 10+ partitions per topic the number of stores can get >> out of hand pretty easily. >> Or did I miss something obvious how those can be monitored better? >> >> best regards >> >> Patrik >> >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna wrote: >>> >>> Hi all, >>> >>> this KIP describes the extension of the Kafka Streams' metrics to include >>> RocksDB's internal statistics. >>> >>> Please have a look at it and let me know what you think. Since I am not a >>> RocksDB expert, I am thankful for any additional pair of eyes that >>> evaluates this KIP. >>> >>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams >>> >>> Best regards, >>> Bruno >>> >>
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Patrik, As of 2.3 you will be able to use the RocksDBConfigSetter to effectively bound the total memory used by RocksDB for a single app instance. You should already be able to limit the memory used per rocksdb store, though as you mention there can be a lot of them. I'm not sure you can monitor the memory usage if you are not limiting it though. On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl wrote: > Hi > Thanks Bruno for the KIP, this is a very good idea. > > I have one question, are there metrics available for the memory consumption > of RocksDB? > As they are running outside the JVM we have run into issues because they > were using all the other memory. > And with multiple streams applications on the same machine, each with > several KTables and 10+ partitions per topic the number of stores can get > out of hand pretty easily. > Or did I miss something obvious how those can be monitored better? > > best regards > > Patrik > > On Fri, 17 May 2019 at 23:54, Bruno Cadonna wrote: > > > Hi all, > > > > this KIP describes the extension of the Kafka Streams' metrics to include > > RocksDB's internal statistics. > > > > Please have a look at it and let me know what you think. Since I am not a > > RocksDB expert, I am thankful for any additional pair of eyes that > > evaluates this KIP. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > > > Best regards, > > Bruno > > >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Thanks Bruno for the KIP, this is a very good idea. I have one question, are there metrics available for the memory consumption of RocksDB? As they are running outside the JVM we have run into issues because they were using all the other memory. And with multiple streams applications on the same machine, each with several KTables and 10+ partitions per topic the number of stores can get out of hand pretty easily. Or did I miss something obvious how those can be monitored better? best regards Patrik On Fri, 17 May 2019 at 23:54, Bruno Cadonna wrote: > Hi all, > > this KIP describes the extension of the Kafka Streams' metrics to include > RocksDB's internal statistics. > > Please have a look at it and let me know what you think. Since I am not a > RocksDB expert, I am thankful for any additional pair of eyes that > evaluates this KIP. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > Best regards, > Bruno >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
I'm not sure we can safely assume only the most recent segment is hot. Anything within the current window size is still being actively queried, and users can independently set windowSize and retentionPeriod as long as windowSize <= retentionPeriod. But the default segmentInterval is max(retentionPeriod / 2, 60,000 ms). So if a store had windowSize ≈ retentionPeriod then the current window actually spans the latest two or three stores. We could consider just providing metrics for however many stores fit within windowSize but that will basically be either all/most of them, or leave out anything being queried by IQ. Of course the IQ access patterns are probably fairly different so there's some question of whether they should be included in the metrics, but I don't see how you could get rocksdb to distinguish which reads are coming from Streams and which are IQ. You could trim some of them by only reporting metrics for stores within the windowSize as mentioned above, but then you'd still be mixing in IQ that just happened to be in the latest stores. It's probably better to include all IQ or none, and I'm not sure how you could accomplish none. On Thu, Jun 6, 2019 at 12:14 PM Bruno Cadonna wrote: > Hi, > > I like the idea of just exposing the metrics of the latest segment. I > think it gives the most realistic picture of the current operations on > the segmented RocksDB without exposing implementation details. The > cons of this approach is that during the switch to a new segment the > values of some metrics might behave a bit strangely because of the > empty memtable and empty caches of the new segment. I will follow up > on this idea and see what I need to change in the KIP. > > Best, > Bruno > > > > > On Thu, Jun 6, 2019 at 2:33 AM Guozhang Wang wrote: > > > > I think Bruno's 2) is that for a segmented store, the access rate on > > different segments will very likely be different. And in fact, most of > the > > access should be on the "latest" segment unless 1) very late arrived > data, > > which should be captured on the higher-level `lateness` metrics already, > > and 2) IQ reads on old windows. The problem is that, say if 99% of reads > go > > to the latest segment, and 1% goes to rest of the segments, how should > > `memtable-hit-rate` be calculated then. > > > > Another wild thought just to throw here: maybe we can just expose the > > latest segment's state store as the logical store's metrics? Admittedly > it > > would not be most accurate, but it is 1) future-proof if we want to > > consolidate to 1-1 physical store -> logical store implementation, and 2) > > it is as simple and not needing to bookkeep older segments who should be > > rarely accessed. My question is though, if upon segment rolling our > metrics > > can be auto-switched to the new store. > > > > > > Guozhang > > > > On Tue, Jun 4, 2019 at 3:06 PM Sophie Blee-Goldman > > wrote: > > > > > Hey Bruno, > > > > > > I tend to agree with Guozhang on this matter although you do bring up > some > > > good points that should be addressed. Regarding 1) I think it is > probably > > > fairly uncommon in practice for users to leverage the individual store > > > names passed to RocksDBConfigSetter#setConfig in order to specify > options > > > on a per-store basis. When this actually is used, it does seem likely > that > > > users would be doing something like pattern matching the physical store > > > name prefix in order to apply configs to all physical stores (segments) > > > within a single logical RocksDBStore. As you mention this is something > of a > > > hassle already as physical stores are created/deleted, while most > likely > > > all anyone cares about is the prefix corresponding to the logical > store. It > > > seems like rather than persist this hassle to the metric layer, we > should > > > consider refactoring RocksDBConfigSetter to apply to a logical store > rather > > > than a specific physical segment. Or maybe providing some kind of > tooling > > > to at least make this easier on users, but that's definitely outside > the > > > scope of this KIP. > > > > > > Regarding 2) can you clarify your point about accessing stores > uniformly? > > > While I agree there will definitely be variance in the access pattern > of > > > different segments, I think it's unlikely that it will vary in any > kind of > > > predictable or deterministic way, hence it is not that useful to know > in > > > hindsight the difference reflected by the metrics. > > > > > > Cheers, > > > Sophie > > > > > > On Tue, Jun 4, 2019 at 2:09 PM Bruno Cadonna > wrote: > > > > > > > Hi Guozhang, > > > > > > > > After some thoughts, I tend to be in favour of the option with > metrics > > > > for each physical RocksDB instance for the following reasons: > > > > > > > > 1) A user already needs to be aware of segmented state stores when > > > > providing a custom RocksDBConfigSetter. In RocksDBConfigSetter one > can > > > > specify settings for a store depending on the name o
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi, I like the idea of just exposing the metrics of the latest segment. I think it gives the most realistic picture of the current operations on the segmented RocksDB without exposing implementation details. The cons of this approach is that during the switch to a new segment the values of some metrics might behave a bit strangely because of the empty memtable and empty caches of the new segment. I will follow up on this idea and see what I need to change in the KIP. Best, Bruno On Thu, Jun 6, 2019 at 2:33 AM Guozhang Wang wrote: > > I think Bruno's 2) is that for a segmented store, the access rate on > different segments will very likely be different. And in fact, most of the > access should be on the "latest" segment unless 1) very late arrived data, > which should be captured on the higher-level `lateness` metrics already, > and 2) IQ reads on old windows. The problem is that, say if 99% of reads go > to the latest segment, and 1% goes to rest of the segments, how should > `memtable-hit-rate` be calculated then. > > Another wild thought just to throw here: maybe we can just expose the > latest segment's state store as the logical store's metrics? Admittedly it > would not be most accurate, but it is 1) future-proof if we want to > consolidate to 1-1 physical store -> logical store implementation, and 2) > it is as simple and not needing to bookkeep older segments who should be > rarely accessed. My question is though, if upon segment rolling our metrics > can be auto-switched to the new store. > > > Guozhang > > On Tue, Jun 4, 2019 at 3:06 PM Sophie Blee-Goldman > wrote: > > > Hey Bruno, > > > > I tend to agree with Guozhang on this matter although you do bring up some > > good points that should be addressed. Regarding 1) I think it is probably > > fairly uncommon in practice for users to leverage the individual store > > names passed to RocksDBConfigSetter#setConfig in order to specify options > > on a per-store basis. When this actually is used, it does seem likely that > > users would be doing something like pattern matching the physical store > > name prefix in order to apply configs to all physical stores (segments) > > within a single logical RocksDBStore. As you mention this is something of a > > hassle already as physical stores are created/deleted, while most likely > > all anyone cares about is the prefix corresponding to the logical store. It > > seems like rather than persist this hassle to the metric layer, we should > > consider refactoring RocksDBConfigSetter to apply to a logical store rather > > than a specific physical segment. Or maybe providing some kind of tooling > > to at least make this easier on users, but that's definitely outside the > > scope of this KIP. > > > > Regarding 2) can you clarify your point about accessing stores uniformly? > > While I agree there will definitely be variance in the access pattern of > > different segments, I think it's unlikely that it will vary in any kind of > > predictable or deterministic way, hence it is not that useful to know in > > hindsight the difference reflected by the metrics. > > > > Cheers, > > Sophie > > > > On Tue, Jun 4, 2019 at 2:09 PM Bruno Cadonna wrote: > > > > > Hi Guozhang, > > > > > > After some thoughts, I tend to be in favour of the option with metrics > > > for each physical RocksDB instance for the following reasons: > > > > > > 1) A user already needs to be aware of segmented state stores when > > > providing a custom RocksDBConfigSetter. In RocksDBConfigSetter one can > > > specify settings for a store depending on the name of the store. Since > > > segments (i.e. state store) in a segmented state store have names that > > > share a prefix but have suffixes that are created at runtime, increase > > > with time and are theoretically unbounded, a user needs to take > > > account of the segments to provide the settings for all (i.e. matching > > > the common prefix) or some (i.e. matching the common prefix and for > > > example suffixes according to a specific pattern) of the segments of a > > > specific segmented state store. > > > 2) Currently settings for RocksDB can only be specified by a user per > > > physical instance and not per logical instance. Deriving good settings > > > for physical instances from metrics for a logical instance can be hard > > > if the physical instances are not accessed uniformly. In segmented > > > state stores segments are not accessed uniformly. > > > 3) Simpler to implement and to get things done. > > > > > > Any thoughts on this from anybody? > > > > > > Best, > > > Bruno > > > > > > On Thu, May 30, 2019 at 8:33 PM Guozhang Wang > > wrote: > > > > > > > > Hi Bruno: > > > > > > > > Regarding 2) I think either way has some shortcomings: exposing the > > > metrics > > > > per rocksDB instance for window / session stores exposed some > > > > implementation internals (that we use segmented stores) to enforce > > users > > > to > > > > be aware of them. E.g. what if we want to si
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
I think Bruno's 2) is that for a segmented store, the access rate on different segments will very likely be different. And in fact, most of the access should be on the "latest" segment unless 1) very late arrived data, which should be captured on the higher-level `lateness` metrics already, and 2) IQ reads on old windows. The problem is that, say if 99% of reads go to the latest segment, and 1% goes to rest of the segments, how should `memtable-hit-rate` be calculated then. Another wild thought just to throw here: maybe we can just expose the latest segment's state store as the logical store's metrics? Admittedly it would not be most accurate, but it is 1) future-proof if we want to consolidate to 1-1 physical store -> logical store implementation, and 2) it is as simple and not needing to bookkeep older segments who should be rarely accessed. My question is though, if upon segment rolling our metrics can be auto-switched to the new store. Guozhang On Tue, Jun 4, 2019 at 3:06 PM Sophie Blee-Goldman wrote: > Hey Bruno, > > I tend to agree with Guozhang on this matter although you do bring up some > good points that should be addressed. Regarding 1) I think it is probably > fairly uncommon in practice for users to leverage the individual store > names passed to RocksDBConfigSetter#setConfig in order to specify options > on a per-store basis. When this actually is used, it does seem likely that > users would be doing something like pattern matching the physical store > name prefix in order to apply configs to all physical stores (segments) > within a single logical RocksDBStore. As you mention this is something of a > hassle already as physical stores are created/deleted, while most likely > all anyone cares about is the prefix corresponding to the logical store. It > seems like rather than persist this hassle to the metric layer, we should > consider refactoring RocksDBConfigSetter to apply to a logical store rather > than a specific physical segment. Or maybe providing some kind of tooling > to at least make this easier on users, but that's definitely outside the > scope of this KIP. > > Regarding 2) can you clarify your point about accessing stores uniformly? > While I agree there will definitely be variance in the access pattern of > different segments, I think it's unlikely that it will vary in any kind of > predictable or deterministic way, hence it is not that useful to know in > hindsight the difference reflected by the metrics. > > Cheers, > Sophie > > On Tue, Jun 4, 2019 at 2:09 PM Bruno Cadonna wrote: > > > Hi Guozhang, > > > > After some thoughts, I tend to be in favour of the option with metrics > > for each physical RocksDB instance for the following reasons: > > > > 1) A user already needs to be aware of segmented state stores when > > providing a custom RocksDBConfigSetter. In RocksDBConfigSetter one can > > specify settings for a store depending on the name of the store. Since > > segments (i.e. state store) in a segmented state store have names that > > share a prefix but have suffixes that are created at runtime, increase > > with time and are theoretically unbounded, a user needs to take > > account of the segments to provide the settings for all (i.e. matching > > the common prefix) or some (i.e. matching the common prefix and for > > example suffixes according to a specific pattern) of the segments of a > > specific segmented state store. > > 2) Currently settings for RocksDB can only be specified by a user per > > physical instance and not per logical instance. Deriving good settings > > for physical instances from metrics for a logical instance can be hard > > if the physical instances are not accessed uniformly. In segmented > > state stores segments are not accessed uniformly. > > 3) Simpler to implement and to get things done. > > > > Any thoughts on this from anybody? > > > > Best, > > Bruno > > > > On Thu, May 30, 2019 at 8:33 PM Guozhang Wang > wrote: > > > > > > Hi Bruno: > > > > > > Regarding 2) I think either way has some shortcomings: exposing the > > metrics > > > per rocksDB instance for window / session stores exposed some > > > implementation internals (that we use segmented stores) to enforce > users > > to > > > be aware of them. E.g. what if we want to silently change the internal > > > implementation by walking away from the segmented approach? On the > other > > > hand, coalescing multiple rocksDB instances' metrics into a single one > > per > > > each logical store also has some concerns as I mentioned above. What > I'm > > > thinking is actually that, if we can customize the aggregation logic to > > > still has one set of metrics per each logical store which may be > composed > > > of multiple rocksDB ones -- e.g. for `bytes-written-rate` we sum them > > > across rocksDBs, while for `memtable-hit-rate` we do weighted average? > > > > > > Regarding logging levels, I think have DEBUG is fine, but also that > means > > > without manually turning it on users wo
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hey Bruno, I tend to agree with Guozhang on this matter although you do bring up some good points that should be addressed. Regarding 1) I think it is probably fairly uncommon in practice for users to leverage the individual store names passed to RocksDBConfigSetter#setConfig in order to specify options on a per-store basis. When this actually is used, it does seem likely that users would be doing something like pattern matching the physical store name prefix in order to apply configs to all physical stores (segments) within a single logical RocksDBStore. As you mention this is something of a hassle already as physical stores are created/deleted, while most likely all anyone cares about is the prefix corresponding to the logical store. It seems like rather than persist this hassle to the metric layer, we should consider refactoring RocksDBConfigSetter to apply to a logical store rather than a specific physical segment. Or maybe providing some kind of tooling to at least make this easier on users, but that's definitely outside the scope of this KIP. Regarding 2) can you clarify your point about accessing stores uniformly? While I agree there will definitely be variance in the access pattern of different segments, I think it's unlikely that it will vary in any kind of predictable or deterministic way, hence it is not that useful to know in hindsight the difference reflected by the metrics. Cheers, Sophie On Tue, Jun 4, 2019 at 2:09 PM Bruno Cadonna wrote: > Hi Guozhang, > > After some thoughts, I tend to be in favour of the option with metrics > for each physical RocksDB instance for the following reasons: > > 1) A user already needs to be aware of segmented state stores when > providing a custom RocksDBConfigSetter. In RocksDBConfigSetter one can > specify settings for a store depending on the name of the store. Since > segments (i.e. state store) in a segmented state store have names that > share a prefix but have suffixes that are created at runtime, increase > with time and are theoretically unbounded, a user needs to take > account of the segments to provide the settings for all (i.e. matching > the common prefix) or some (i.e. matching the common prefix and for > example suffixes according to a specific pattern) of the segments of a > specific segmented state store. > 2) Currently settings for RocksDB can only be specified by a user per > physical instance and not per logical instance. Deriving good settings > for physical instances from metrics for a logical instance can be hard > if the physical instances are not accessed uniformly. In segmented > state stores segments are not accessed uniformly. > 3) Simpler to implement and to get things done. > > Any thoughts on this from anybody? > > Best, > Bruno > > On Thu, May 30, 2019 at 8:33 PM Guozhang Wang wrote: > > > > Hi Bruno: > > > > Regarding 2) I think either way has some shortcomings: exposing the > metrics > > per rocksDB instance for window / session stores exposed some > > implementation internals (that we use segmented stores) to enforce users > to > > be aware of them. E.g. what if we want to silently change the internal > > implementation by walking away from the segmented approach? On the other > > hand, coalescing multiple rocksDB instances' metrics into a single one > per > > each logical store also has some concerns as I mentioned above. What I'm > > thinking is actually that, if we can customize the aggregation logic to > > still has one set of metrics per each logical store which may be composed > > of multiple rocksDB ones -- e.g. for `bytes-written-rate` we sum them > > across rocksDBs, while for `memtable-hit-rate` we do weighted average? > > > > Regarding logging levels, I think have DEBUG is fine, but also that means > > without manually turning it on users would not get it. Maybe we should > > consider adding some dynamic setting mechanisms in the future to allow > > users turn it on / off during run-time. > > > > > > Guozhang > > > > > > > > On Tue, May 28, 2019 at 6:23 AM Bruno Cadonna > wrote: > > > > > Hi, > > > > > > Thank you for your comments. > > > > > > @Bill: > > > > > > 1. It is like Guozhang wrote: > > > - rocksdb-state-id is for key-value stores > > > - rocksdb-session-state-id is for session stores > > > - rocksdb-window-state-id is for window stores > > > These tags are defined in the corresponding store builders and I think > > > it is a good idea to re-use them. > > > > > > 2. I could not find any exposed ticker or histogram to get the total > > > and average number of compactions, although RocksDB dumps the number > > > of compactions between levels in its log files. There is the > > > NUM_SUBCOMPACTIONS_SCHEDULED histogram that gives you statistics about > > > the number of subcompactions actually scheduled during a compaction, > > > but that is not that what you are looking for. If they will expose the > > > number of compaction in the future, we can still add the metrics you > > > propose. I guess, the
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Guozhang, After some thoughts, I tend to be in favour of the option with metrics for each physical RocksDB instance for the following reasons: 1) A user already needs to be aware of segmented state stores when providing a custom RocksDBConfigSetter. In RocksDBConfigSetter one can specify settings for a store depending on the name of the store. Since segments (i.e. state store) in a segmented state store have names that share a prefix but have suffixes that are created at runtime, increase with time and are theoretically unbounded, a user needs to take account of the segments to provide the settings for all (i.e. matching the common prefix) or some (i.e. matching the common prefix and for example suffixes according to a specific pattern) of the segments of a specific segmented state store. 2) Currently settings for RocksDB can only be specified by a user per physical instance and not per logical instance. Deriving good settings for physical instances from metrics for a logical instance can be hard if the physical instances are not accessed uniformly. In segmented state stores segments are not accessed uniformly. 3) Simpler to implement and to get things done. Any thoughts on this from anybody? Best, Bruno On Thu, May 30, 2019 at 8:33 PM Guozhang Wang wrote: > > Hi Bruno: > > Regarding 2) I think either way has some shortcomings: exposing the metrics > per rocksDB instance for window / session stores exposed some > implementation internals (that we use segmented stores) to enforce users to > be aware of them. E.g. what if we want to silently change the internal > implementation by walking away from the segmented approach? On the other > hand, coalescing multiple rocksDB instances' metrics into a single one per > each logical store also has some concerns as I mentioned above. What I'm > thinking is actually that, if we can customize the aggregation logic to > still has one set of metrics per each logical store which may be composed > of multiple rocksDB ones -- e.g. for `bytes-written-rate` we sum them > across rocksDBs, while for `memtable-hit-rate` we do weighted average? > > Regarding logging levels, I think have DEBUG is fine, but also that means > without manually turning it on users would not get it. Maybe we should > consider adding some dynamic setting mechanisms in the future to allow > users turn it on / off during run-time. > > > Guozhang > > > > On Tue, May 28, 2019 at 6:23 AM Bruno Cadonna wrote: > > > Hi, > > > > Thank you for your comments. > > > > @Bill: > > > > 1. It is like Guozhang wrote: > > - rocksdb-state-id is for key-value stores > > - rocksdb-session-state-id is for session stores > > - rocksdb-window-state-id is for window stores > > These tags are defined in the corresponding store builders and I think > > it is a good idea to re-use them. > > > > 2. I could not find any exposed ticker or histogram to get the total > > and average number of compactions, although RocksDB dumps the number > > of compactions between levels in its log files. There is the > > NUM_SUBCOMPACTIONS_SCHEDULED histogram that gives you statistics about > > the number of subcompactions actually scheduled during a compaction, > > but that is not that what you are looking for. If they will expose the > > number of compaction in the future, we can still add the metrics you > > propose. I guess, the metric in this KIP that would indirectly be > > influenced by the number of L0 files would be write-stall-duration. If > > there are too many compactions this duration should increase. However, > > this metric is also influenced by memtable flushes. > > > > @John: > > > > I think it is a good idea to prefix the flush-related metrics with > > memtable to avoid ambiguity. I changed the KIP accordingly. > > > > @Dongjin: > > > > For the tag and compaction-related comments, please see my answers to Bill. > > > > I cannot follow your second paragraph. Are you saying that a tuning > > guide for RocksDB within Streams based on the metrics in this KIP is > > out of scope? I also think that it doesn't need to be included in this > > KIP, but it is worth to work on it afterwards. > > > > @Guozhang: > > > > 1. Thank you for the explanation. I missed that. I modified the KIP > > accordingly. > > > > 2. No, my plan is to record and expose the set of metrics for each > > RocksDB store separately. Each set of metrics can then be > > distinguished by its store ID. Do I miss something here? > > > > 3. I agree with you and Sophie about user education and that we should > > work on it after this KIP. > > > > 4. I agree also on the user API. However, I would like to open a > > separate KIP for it because I still need a bit of thinking to get it. > > I also think it is a good idea to do one step after the other to get > > at least the built-in RocksDB metrics into the next release. > > Do you think I chose too many metrics as built-in metrics for this > > KIP? What do others think? > > > > @Sophie: > > > > I tend to DEBUG level, bu
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Bruno: Regarding 2) I think either way has some shortcomings: exposing the metrics per rocksDB instance for window / session stores exposed some implementation internals (that we use segmented stores) to enforce users to be aware of them. E.g. what if we want to silently change the internal implementation by walking away from the segmented approach? On the other hand, coalescing multiple rocksDB instances' metrics into a single one per each logical store also has some concerns as I mentioned above. What I'm thinking is actually that, if we can customize the aggregation logic to still has one set of metrics per each logical store which may be composed of multiple rocksDB ones -- e.g. for `bytes-written-rate` we sum them across rocksDBs, while for `memtable-hit-rate` we do weighted average? Regarding logging levels, I think have DEBUG is fine, but also that means without manually turning it on users would not get it. Maybe we should consider adding some dynamic setting mechanisms in the future to allow users turn it on / off during run-time. Guozhang On Tue, May 28, 2019 at 6:23 AM Bruno Cadonna wrote: > Hi, > > Thank you for your comments. > > @Bill: > > 1. It is like Guozhang wrote: > - rocksdb-state-id is for key-value stores > - rocksdb-session-state-id is for session stores > - rocksdb-window-state-id is for window stores > These tags are defined in the corresponding store builders and I think > it is a good idea to re-use them. > > 2. I could not find any exposed ticker or histogram to get the total > and average number of compactions, although RocksDB dumps the number > of compactions between levels in its log files. There is the > NUM_SUBCOMPACTIONS_SCHEDULED histogram that gives you statistics about > the number of subcompactions actually scheduled during a compaction, > but that is not that what you are looking for. If they will expose the > number of compaction in the future, we can still add the metrics you > propose. I guess, the metric in this KIP that would indirectly be > influenced by the number of L0 files would be write-stall-duration. If > there are too many compactions this duration should increase. However, > this metric is also influenced by memtable flushes. > > @John: > > I think it is a good idea to prefix the flush-related metrics with > memtable to avoid ambiguity. I changed the KIP accordingly. > > @Dongjin: > > For the tag and compaction-related comments, please see my answers to Bill. > > I cannot follow your second paragraph. Are you saying that a tuning > guide for RocksDB within Streams based on the metrics in this KIP is > out of scope? I also think that it doesn't need to be included in this > KIP, but it is worth to work on it afterwards. > > @Guozhang: > > 1. Thank you for the explanation. I missed that. I modified the KIP > accordingly. > > 2. No, my plan is to record and expose the set of metrics for each > RocksDB store separately. Each set of metrics can then be > distinguished by its store ID. Do I miss something here? > > 3. I agree with you and Sophie about user education and that we should > work on it after this KIP. > > 4. I agree also on the user API. However, I would like to open a > separate KIP for it because I still need a bit of thinking to get it. > I also think it is a good idea to do one step after the other to get > at least the built-in RocksDB metrics into the next release. > Do you think I chose too many metrics as built-in metrics for this > KIP? What do others think? > > @Sophie: > > I tend to DEBUG level, but I do not have heart feelings about it. Do > you mean to turn it on/off RocksDB metrics in the Streams > configuration? > > Best, > Bruno > > On Tue, May 21, 2019 at 8:02 PM Sophie Blee-Goldman > wrote: > > > > I definitely agree with Guozhang's "meta" comment: if it's possible to > > allow users to pick and choose individual RocksDB metrics that would be > > ideal. One further question is whether these will be debug or info level > > metrics, or a separate level altogether? If there is a nontrivial > overhead > > associated with attaching RocksDB metrics it would probably be good to be > > able to independently turn on/off Rocks metrics > > > > On Tue, May 21, 2019 at 9:00 AM Guozhang Wang > wrote: > > > > > Hello Bruno, > > > > > > Thanks for the KIP, I have a few minor comments and a meta one which > are > > > relatively aligned with other folks: > > > > > > Minor: > > > > > > 1) Regarding the "rocksdb-state-id = [store ID]", to be consistent with > > > other state store metrics (see > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams > ), > > > this tag should be either "rocksdb-window-state-id", > > > "rocksdb-session-state-id" or "rocksdb-state-id". For window / session > > > store backed by rocksDB, the tags should not be "rocksdb-state-id". > > > > > > 2) Also for window / session store, my take is that you plan to have > > > multiple rocksDB behind the scene to report
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi, Thank you for your comments. @Bill: 1. It is like Guozhang wrote: - rocksdb-state-id is for key-value stores - rocksdb-session-state-id is for session stores - rocksdb-window-state-id is for window stores These tags are defined in the corresponding store builders and I think it is a good idea to re-use them. 2. I could not find any exposed ticker or histogram to get the total and average number of compactions, although RocksDB dumps the number of compactions between levels in its log files. There is the NUM_SUBCOMPACTIONS_SCHEDULED histogram that gives you statistics about the number of subcompactions actually scheduled during a compaction, but that is not that what you are looking for. If they will expose the number of compaction in the future, we can still add the metrics you propose. I guess, the metric in this KIP that would indirectly be influenced by the number of L0 files would be write-stall-duration. If there are too many compactions this duration should increase. However, this metric is also influenced by memtable flushes. @John: I think it is a good idea to prefix the flush-related metrics with memtable to avoid ambiguity. I changed the KIP accordingly. @Dongjin: For the tag and compaction-related comments, please see my answers to Bill. I cannot follow your second paragraph. Are you saying that a tuning guide for RocksDB within Streams based on the metrics in this KIP is out of scope? I also think that it doesn't need to be included in this KIP, but it is worth to work on it afterwards. @Guozhang: 1. Thank you for the explanation. I missed that. I modified the KIP accordingly. 2. No, my plan is to record and expose the set of metrics for each RocksDB store separately. Each set of metrics can then be distinguished by its store ID. Do I miss something here? 3. I agree with you and Sophie about user education and that we should work on it after this KIP. 4. I agree also on the user API. However, I would like to open a separate KIP for it because I still need a bit of thinking to get it. I also think it is a good idea to do one step after the other to get at least the built-in RocksDB metrics into the next release. Do you think I chose too many metrics as built-in metrics for this KIP? What do others think? @Sophie: I tend to DEBUG level, but I do not have heart feelings about it. Do you mean to turn it on/off RocksDB metrics in the Streams configuration? Best, Bruno On Tue, May 21, 2019 at 8:02 PM Sophie Blee-Goldman wrote: > > I definitely agree with Guozhang's "meta" comment: if it's possible to > allow users to pick and choose individual RocksDB metrics that would be > ideal. One further question is whether these will be debug or info level > metrics, or a separate level altogether? If there is a nontrivial overhead > associated with attaching RocksDB metrics it would probably be good to be > able to independently turn on/off Rocks metrics > > On Tue, May 21, 2019 at 9:00 AM Guozhang Wang wrote: > > > Hello Bruno, > > > > Thanks for the KIP, I have a few minor comments and a meta one which are > > relatively aligned with other folks: > > > > Minor: > > > > 1) Regarding the "rocksdb-state-id = [store ID]", to be consistent with > > other state store metrics (see > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams), > > this tag should be either "rocksdb-window-state-id", > > "rocksdb-session-state-id" or "rocksdb-state-id". For window / session > > store backed by rocksDB, the tags should not be "rocksdb-state-id". > > > > 2) Also for window / session store, my take is that you plan to have > > multiple rocksDB behind the scene to report to the same set of metrics, is > > that right? My concern is that for such types of state stores, most of the > > access would be on the first segment rocksDB instance, and hence coalescing > > all of instances as a single set of metrics may dilute it. > > > > 3) I agree with @sop...@confluent.io that we should > > better have some documentation educating users what to do when see what > > anomalies in metrics; though I think this is a long endeavoring effort that > > may go beyond this KIP's scope, let's keep that as a separate umbrella JIRA > > to accumulate knowledge over time. > > > > > > Meta: > > > > 4) Instead of trying to enumerate all the ones that might be helpful, I'd > > recommend that we expose a user-friendly API in StreamsMetrics to allow > > users to add more metrics from RocksDB they'd like to have, while only > > keeping a small set of most-meaningful ones that are ubiquitously useful > > out-of-the-box. WDYT? > > > > > > Guozhang > > > > > > > > On Tue, May 21, 2019 at 8:04 AM Dongjin Lee wrote: > > > >> Hi Bruno, > >> > >> I just read the KIP. I think this feature is great. As far as I know, most > >> Kafka users monitor the host resources, JVM resources, and Kafka metrics > >> only, not RocksDB for configuring the statistics feature is a little bit > >> tiresome. Since
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
I definitely agree with Guozhang's "meta" comment: if it's possible to allow users to pick and choose individual RocksDB metrics that would be ideal. One further question is whether these will be debug or info level metrics, or a separate level altogether? If there is a nontrivial overhead associated with attaching RocksDB metrics it would probably be good to be able to independently turn on/off Rocks metrics On Tue, May 21, 2019 at 9:00 AM Guozhang Wang wrote: > Hello Bruno, > > Thanks for the KIP, I have a few minor comments and a meta one which are > relatively aligned with other folks: > > Minor: > > 1) Regarding the "rocksdb-state-id = [store ID]", to be consistent with > other state store metrics (see > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams), > this tag should be either "rocksdb-window-state-id", > "rocksdb-session-state-id" or "rocksdb-state-id". For window / session > store backed by rocksDB, the tags should not be "rocksdb-state-id". > > 2) Also for window / session store, my take is that you plan to have > multiple rocksDB behind the scene to report to the same set of metrics, is > that right? My concern is that for such types of state stores, most of the > access would be on the first segment rocksDB instance, and hence coalescing > all of instances as a single set of metrics may dilute it. > > 3) I agree with @sop...@confluent.io that we should > better have some documentation educating users what to do when see what > anomalies in metrics; though I think this is a long endeavoring effort that > may go beyond this KIP's scope, let's keep that as a separate umbrella JIRA > to accumulate knowledge over time. > > > Meta: > > 4) Instead of trying to enumerate all the ones that might be helpful, I'd > recommend that we expose a user-friendly API in StreamsMetrics to allow > users to add more metrics from RocksDB they'd like to have, while only > keeping a small set of most-meaningful ones that are ubiquitously useful > out-of-the-box. WDYT? > > > Guozhang > > > > On Tue, May 21, 2019 at 8:04 AM Dongjin Lee wrote: > >> Hi Bruno, >> >> I just read the KIP. I think this feature is great. As far as I know, most >> Kafka users monitor the host resources, JVM resources, and Kafka metrics >> only, not RocksDB for configuring the statistics feature is a little bit >> tiresome. Since RocksDB impacts the performance of Kafka Streams, I >> believe >> providing these metrics with other metrics in one place is much better. >> >> However, I am a little bit not assured for how much information should be >> provided to the users with the documentation - how the user can control >> the >> RocksDB may on the boundary of the scope. How do you think? >> >> +1. I entirely agree to Bill's comments; I also think `rocksdb-store-id` >> is >> better than `rocksdb-state-id` and metrics on total compactions and an >> average number of compactions is also needed. >> >> Regards, >> Dongjin >> >> On Tue, May 21, 2019 at 2:48 AM John Roesler wrote: >> >> > Hi Bruno, >> > >> > Looks really good overall. This is going to be an awesome addition. >> > >> > My only thought was that we have "bytes-flushed-(rate|total) and >> > flush-time-(avg|min|max)" metrics, and the description states that >> > these are specifically for Memtable flush operations. What do you >> > think about calling it "memtable-bytes-flushed... (etc)"? On one hand, >> > I could see this being redundant, since that's the only thing that >> > gets "flushed" inside of Rocks. But on the other, we have an >> > independent "flush" operation in streams, which might be confusing. >> > Plus, it might help people who are looking at the full "menu" of >> > hundreds of metrics. They can't read and remember every description >> > while trying to understand the full list of metrics, so going for >> > maximum self-documentation in the name seems nice. >> > >> > But that's a minor thought. Modulo the others' comments, this looks good >> > to me. >> > >> > Thanks, >> > -John >> > >> > On Mon, May 20, 2019 at 11:07 AM Bill Bejeck wrote: >> > > >> > > Hi Bruno, >> > > >> > > Thanks for the KIP, this will be a useful addition. >> > > >> > > Overall the KIP looks good to me, and I have two minor comments. >> > > >> > > 1. For the tags should, I'm wondering if rocksdb-state-id should be >> > > rocksdb-store-id >> > > instead? >> > > >> > > 2. With the compaction metrics, would it be possible to add total >> > > compactions and an average number of compactions? I've taken a look >> at >> > the >> > > available RocksDB metrics, and I'm not sure. But users can control >> how >> > > many L0 files it takes to trigger compaction so if it is possible; it >> may >> > > be useful. >> > > >> > > Thanks, >> > > Bill >> > > >> > > >> > > On Mon, May 20, 2019 at 9:15 AM Bruno Cadonna >> > wrote: >> > > >> > > > Hi Sophie, >> > > > >> > > > Thank you for your comments. >> > > > >> > > > It's a good idea to supplement the metrics with
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hello Bruno, Thanks for the KIP, I have a few minor comments and a meta one which are relatively aligned with other folks: Minor: 1) Regarding the "rocksdb-state-id = [store ID]", to be consistent with other state store metrics (see https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams), this tag should be either "rocksdb-window-state-id", "rocksdb-session-state-id" or "rocksdb-state-id". For window / session store backed by rocksDB, the tags should not be "rocksdb-state-id". 2) Also for window / session store, my take is that you plan to have multiple rocksDB behind the scene to report to the same set of metrics, is that right? My concern is that for such types of state stores, most of the access would be on the first segment rocksDB instance, and hence coalescing all of instances as a single set of metrics may dilute it. 3) I agree with @sop...@confluent.io that we should better have some documentation educating users what to do when see what anomalies in metrics; though I think this is a long endeavoring effort that may go beyond this KIP's scope, let's keep that as a separate umbrella JIRA to accumulate knowledge over time. Meta: 4) Instead of trying to enumerate all the ones that might be helpful, I'd recommend that we expose a user-friendly API in StreamsMetrics to allow users to add more metrics from RocksDB they'd like to have, while only keeping a small set of most-meaningful ones that are ubiquitously useful out-of-the-box. WDYT? Guozhang On Tue, May 21, 2019 at 8:04 AM Dongjin Lee wrote: > Hi Bruno, > > I just read the KIP. I think this feature is great. As far as I know, most > Kafka users monitor the host resources, JVM resources, and Kafka metrics > only, not RocksDB for configuring the statistics feature is a little bit > tiresome. Since RocksDB impacts the performance of Kafka Streams, I believe > providing these metrics with other metrics in one place is much better. > > However, I am a little bit not assured for how much information should be > provided to the users with the documentation - how the user can control the > RocksDB may on the boundary of the scope. How do you think? > > +1. I entirely agree to Bill's comments; I also think `rocksdb-store-id` is > better than `rocksdb-state-id` and metrics on total compactions and an > average number of compactions is also needed. > > Regards, > Dongjin > > On Tue, May 21, 2019 at 2:48 AM John Roesler wrote: > > > Hi Bruno, > > > > Looks really good overall. This is going to be an awesome addition. > > > > My only thought was that we have "bytes-flushed-(rate|total) and > > flush-time-(avg|min|max)" metrics, and the description states that > > these are specifically for Memtable flush operations. What do you > > think about calling it "memtable-bytes-flushed... (etc)"? On one hand, > > I could see this being redundant, since that's the only thing that > > gets "flushed" inside of Rocks. But on the other, we have an > > independent "flush" operation in streams, which might be confusing. > > Plus, it might help people who are looking at the full "menu" of > > hundreds of metrics. They can't read and remember every description > > while trying to understand the full list of metrics, so going for > > maximum self-documentation in the name seems nice. > > > > But that's a minor thought. Modulo the others' comments, this looks good > > to me. > > > > Thanks, > > -John > > > > On Mon, May 20, 2019 at 11:07 AM Bill Bejeck wrote: > > > > > > Hi Bruno, > > > > > > Thanks for the KIP, this will be a useful addition. > > > > > > Overall the KIP looks good to me, and I have two minor comments. > > > > > > 1. For the tags should, I'm wondering if rocksdb-state-id should be > > > rocksdb-store-id > > > instead? > > > > > > 2. With the compaction metrics, would it be possible to add total > > > compactions and an average number of compactions? I've taken a look at > > the > > > available RocksDB metrics, and I'm not sure. But users can control how > > > many L0 files it takes to trigger compaction so if it is possible; it > may > > > be useful. > > > > > > Thanks, > > > Bill > > > > > > > > > On Mon, May 20, 2019 at 9:15 AM Bruno Cadonna > > wrote: > > > > > > > Hi Sophie, > > > > > > > > Thank you for your comments. > > > > > > > > It's a good idea to supplement the metrics with configuration option > > > > to change the metrics. I also had some thoughts about it. However, I > > > > think I need some experimentation to get this right. > > > > > > > > I added the block cache hit rates for index and filter blocks to the > > > > KIP. As far as I understood, they should stay at zero, if users do > not > > > > configure RocksDB to include index and filter blocks into the block > > > > cache. Did you also understand this similarly? I guess also in this > > > > case some experimentation would be good to be sure. > > > > > > > > Best, > > > > Bruno > > > > > > > > > > > > On Sat, May 18, 2019 at 2:29 AM
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Bruno, I just read the KIP. I think this feature is great. As far as I know, most Kafka users monitor the host resources, JVM resources, and Kafka metrics only, not RocksDB for configuring the statistics feature is a little bit tiresome. Since RocksDB impacts the performance of Kafka Streams, I believe providing these metrics with other metrics in one place is much better. However, I am a little bit not assured for how much information should be provided to the users with the documentation - how the user can control the RocksDB may on the boundary of the scope. How do you think? +1. I entirely agree to Bill's comments; I also think `rocksdb-store-id` is better than `rocksdb-state-id` and metrics on total compactions and an average number of compactions is also needed. Regards, Dongjin On Tue, May 21, 2019 at 2:48 AM John Roesler wrote: > Hi Bruno, > > Looks really good overall. This is going to be an awesome addition. > > My only thought was that we have "bytes-flushed-(rate|total) and > flush-time-(avg|min|max)" metrics, and the description states that > these are specifically for Memtable flush operations. What do you > think about calling it "memtable-bytes-flushed... (etc)"? On one hand, > I could see this being redundant, since that's the only thing that > gets "flushed" inside of Rocks. But on the other, we have an > independent "flush" operation in streams, which might be confusing. > Plus, it might help people who are looking at the full "menu" of > hundreds of metrics. They can't read and remember every description > while trying to understand the full list of metrics, so going for > maximum self-documentation in the name seems nice. > > But that's a minor thought. Modulo the others' comments, this looks good > to me. > > Thanks, > -John > > On Mon, May 20, 2019 at 11:07 AM Bill Bejeck wrote: > > > > Hi Bruno, > > > > Thanks for the KIP, this will be a useful addition. > > > > Overall the KIP looks good to me, and I have two minor comments. > > > > 1. For the tags should, I'm wondering if rocksdb-state-id should be > > rocksdb-store-id > > instead? > > > > 2. With the compaction metrics, would it be possible to add total > > compactions and an average number of compactions? I've taken a look at > the > > available RocksDB metrics, and I'm not sure. But users can control how > > many L0 files it takes to trigger compaction so if it is possible; it may > > be useful. > > > > Thanks, > > Bill > > > > > > On Mon, May 20, 2019 at 9:15 AM Bruno Cadonna > wrote: > > > > > Hi Sophie, > > > > > > Thank you for your comments. > > > > > > It's a good idea to supplement the metrics with configuration option > > > to change the metrics. I also had some thoughts about it. However, I > > > think I need some experimentation to get this right. > > > > > > I added the block cache hit rates for index and filter blocks to the > > > KIP. As far as I understood, they should stay at zero, if users do not > > > configure RocksDB to include index and filter blocks into the block > > > cache. Did you also understand this similarly? I guess also in this > > > case some experimentation would be good to be sure. > > > > > > Best, > > > Bruno > > > > > > > > > On Sat, May 18, 2019 at 2:29 AM Sophie Blee-Goldman < > sop...@confluent.io> > > > wrote: > > > > > > > > Actually I wonder if it might be useful to users to be able to break > up > > > the > > > > cache hit stats by type? Some people may choose to store index and > filter > > > > blocks alongside data blocks, and it would probably be very helpful > for > > > > them to know who is making more effective use of the cache in order > to > > > tune > > > > how much of it is allocated to each. I'm not sure how common this > really > > > is > > > > but I think it would be invaluable to those who do. RocksDB > performance > > > can > > > > be quite opaque.. > > > > > > > > Cheers, > > > > Sophie > > > > > > > > On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman < > sop...@confluent.io > > > > > > > > wrote: > > > > > > > > > Hey Bruno! > > > > > > > > > > This all looks pretty good to me, but one suggestion I have is to > > > > > supplement each of the metrics with some info on how the user can > > > control > > > > > them. In other words, which options could/should they set in > > > > > RocksDBConfigSetter should they discover a particular bottleneck? > > > > > > > > > > I don't think this necessarily needs to go into the KIP, but I do > > > think it > > > > > should be included in the docs somewhere (happy to help build up > the > > > list > > > > > of associated options when the time comes) > > > > > > > > > > Thanks! > > > > > Sophie > > > > > > > > > > On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna > > > wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> this KIP describes the extension of the Kafka Streams' metrics to > > > include > > > > >> RocksDB's internal statistics. > > > > >> > > > > >> Please have a look at it and let me know what you think.
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Bruno, Looks really good overall. This is going to be an awesome addition. My only thought was that we have "bytes-flushed-(rate|total) and flush-time-(avg|min|max)" metrics, and the description states that these are specifically for Memtable flush operations. What do you think about calling it "memtable-bytes-flushed... (etc)"? On one hand, I could see this being redundant, since that's the only thing that gets "flushed" inside of Rocks. But on the other, we have an independent "flush" operation in streams, which might be confusing. Plus, it might help people who are looking at the full "menu" of hundreds of metrics. They can't read and remember every description while trying to understand the full list of metrics, so going for maximum self-documentation in the name seems nice. But that's a minor thought. Modulo the others' comments, this looks good to me. Thanks, -John On Mon, May 20, 2019 at 11:07 AM Bill Bejeck wrote: > > Hi Bruno, > > Thanks for the KIP, this will be a useful addition. > > Overall the KIP looks good to me, and I have two minor comments. > > 1. For the tags should, I'm wondering if rocksdb-state-id should be > rocksdb-store-id > instead? > > 2. With the compaction metrics, would it be possible to add total > compactions and an average number of compactions? I've taken a look at the > available RocksDB metrics, and I'm not sure. But users can control how > many L0 files it takes to trigger compaction so if it is possible; it may > be useful. > > Thanks, > Bill > > > On Mon, May 20, 2019 at 9:15 AM Bruno Cadonna wrote: > > > Hi Sophie, > > > > Thank you for your comments. > > > > It's a good idea to supplement the metrics with configuration option > > to change the metrics. I also had some thoughts about it. However, I > > think I need some experimentation to get this right. > > > > I added the block cache hit rates for index and filter blocks to the > > KIP. As far as I understood, they should stay at zero, if users do not > > configure RocksDB to include index and filter blocks into the block > > cache. Did you also understand this similarly? I guess also in this > > case some experimentation would be good to be sure. > > > > Best, > > Bruno > > > > > > On Sat, May 18, 2019 at 2:29 AM Sophie Blee-Goldman > > wrote: > > > > > > Actually I wonder if it might be useful to users to be able to break up > > the > > > cache hit stats by type? Some people may choose to store index and filter > > > blocks alongside data blocks, and it would probably be very helpful for > > > them to know who is making more effective use of the cache in order to > > tune > > > how much of it is allocated to each. I'm not sure how common this really > > is > > > but I think it would be invaluable to those who do. RocksDB performance > > can > > > be quite opaque.. > > > > > > Cheers, > > > Sophie > > > > > > On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman > > > > > wrote: > > > > > > > Hey Bruno! > > > > > > > > This all looks pretty good to me, but one suggestion I have is to > > > > supplement each of the metrics with some info on how the user can > > control > > > > them. In other words, which options could/should they set in > > > > RocksDBConfigSetter should they discover a particular bottleneck? > > > > > > > > I don't think this necessarily needs to go into the KIP, but I do > > think it > > > > should be included in the docs somewhere (happy to help build up the > > list > > > > of associated options when the time comes) > > > > > > > > Thanks! > > > > Sophie > > > > > > > > On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> this KIP describes the extension of the Kafka Streams' metrics to > > include > > > >> RocksDB's internal statistics. > > > >> > > > >> Please have a look at it and let me know what you think. Since I am > > not a > > > >> RocksDB expert, I am thankful for any additional pair of eyes that > > > >> evaluates this KIP. > > > >> > > > >> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > > >> > > > >> Best regards, > > > >> Bruno > > > >> > > > > > >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Bruno, Thanks for the KIP, this will be a useful addition. Overall the KIP looks good to me, and I have two minor comments. 1. For the tags should, I'm wondering if rocksdb-state-id should be rocksdb-store-id instead? 2. With the compaction metrics, would it be possible to add total compactions and an average number of compactions? I've taken a look at the available RocksDB metrics, and I'm not sure. But users can control how many L0 files it takes to trigger compaction so if it is possible; it may be useful. Thanks, Bill On Mon, May 20, 2019 at 9:15 AM Bruno Cadonna wrote: > Hi Sophie, > > Thank you for your comments. > > It's a good idea to supplement the metrics with configuration option > to change the metrics. I also had some thoughts about it. However, I > think I need some experimentation to get this right. > > I added the block cache hit rates for index and filter blocks to the > KIP. As far as I understood, they should stay at zero, if users do not > configure RocksDB to include index and filter blocks into the block > cache. Did you also understand this similarly? I guess also in this > case some experimentation would be good to be sure. > > Best, > Bruno > > > On Sat, May 18, 2019 at 2:29 AM Sophie Blee-Goldman > wrote: > > > > Actually I wonder if it might be useful to users to be able to break up > the > > cache hit stats by type? Some people may choose to store index and filter > > blocks alongside data blocks, and it would probably be very helpful for > > them to know who is making more effective use of the cache in order to > tune > > how much of it is allocated to each. I'm not sure how common this really > is > > but I think it would be invaluable to those who do. RocksDB performance > can > > be quite opaque.. > > > > Cheers, > > Sophie > > > > On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman > > > wrote: > > > > > Hey Bruno! > > > > > > This all looks pretty good to me, but one suggestion I have is to > > > supplement each of the metrics with some info on how the user can > control > > > them. In other words, which options could/should they set in > > > RocksDBConfigSetter should they discover a particular bottleneck? > > > > > > I don't think this necessarily needs to go into the KIP, but I do > think it > > > should be included in the docs somewhere (happy to help build up the > list > > > of associated options when the time comes) > > > > > > Thanks! > > > Sophie > > > > > > On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna > wrote: > > > > > >> Hi all, > > >> > > >> this KIP describes the extension of the Kafka Streams' metrics to > include > > >> RocksDB's internal statistics. > > >> > > >> Please have a look at it and let me know what you think. Since I am > not a > > >> RocksDB expert, I am thankful for any additional pair of eyes that > > >> evaluates this KIP. > > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > >> > > >> Best regards, > > >> Bruno > > >> > > > >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi Sophie, Thank you for your comments. It's a good idea to supplement the metrics with configuration option to change the metrics. I also had some thoughts about it. However, I think I need some experimentation to get this right. I added the block cache hit rates for index and filter blocks to the KIP. As far as I understood, they should stay at zero, if users do not configure RocksDB to include index and filter blocks into the block cache. Did you also understand this similarly? I guess also in this case some experimentation would be good to be sure. Best, Bruno On Sat, May 18, 2019 at 2:29 AM Sophie Blee-Goldman wrote: > > Actually I wonder if it might be useful to users to be able to break up the > cache hit stats by type? Some people may choose to store index and filter > blocks alongside data blocks, and it would probably be very helpful for > them to know who is making more effective use of the cache in order to tune > how much of it is allocated to each. I'm not sure how common this really is > but I think it would be invaluable to those who do. RocksDB performance can > be quite opaque.. > > Cheers, > Sophie > > On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman > wrote: > > > Hey Bruno! > > > > This all looks pretty good to me, but one suggestion I have is to > > supplement each of the metrics with some info on how the user can control > > them. In other words, which options could/should they set in > > RocksDBConfigSetter should they discover a particular bottleneck? > > > > I don't think this necessarily needs to go into the KIP, but I do think it > > should be included in the docs somewhere (happy to help build up the list > > of associated options when the time comes) > > > > Thanks! > > Sophie > > > > On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna wrote: > > > >> Hi all, > >> > >> this KIP describes the extension of the Kafka Streams' metrics to include > >> RocksDB's internal statistics. > >> > >> Please have a look at it and let me know what you think. Since I am not a > >> RocksDB expert, I am thankful for any additional pair of eyes that > >> evaluates this KIP. > >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > >> > >> Best regards, > >> Bruno > >> > >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Actually I wonder if it might be useful to users to be able to break up the cache hit stats by type? Some people may choose to store index and filter blocks alongside data blocks, and it would probably be very helpful for them to know who is making more effective use of the cache in order to tune how much of it is allocated to each. I'm not sure how common this really is but I think it would be invaluable to those who do. RocksDB performance can be quite opaque.. Cheers, Sophie On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman wrote: > Hey Bruno! > > This all looks pretty good to me, but one suggestion I have is to > supplement each of the metrics with some info on how the user can control > them. In other words, which options could/should they set in > RocksDBConfigSetter should they discover a particular bottleneck? > > I don't think this necessarily needs to go into the KIP, but I do think it > should be included in the docs somewhere (happy to help build up the list > of associated options when the time comes) > > Thanks! > Sophie > > On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna wrote: > >> Hi all, >> >> this KIP describes the extension of the Kafka Streams' metrics to include >> RocksDB's internal statistics. >> >> Please have a look at it and let me know what you think. Since I am not a >> RocksDB expert, I am thankful for any additional pair of eyes that >> evaluates this KIP. >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams >> >> Best regards, >> Bruno >> >
Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hey Bruno! This all looks pretty good to me, but one suggestion I have is to supplement each of the metrics with some info on how the user can control them. In other words, which options could/should they set in RocksDBConfigSetter should they discover a particular bottleneck? I don't think this necessarily needs to go into the KIP, but I do think it should be included in the docs somewhere (happy to help build up the list of associated options when the time comes) Thanks! Sophie On Fri, May 17, 2019 at 2:54 PM Bruno Cadonna wrote: > Hi all, > > this KIP describes the extension of the Kafka Streams' metrics to include > RocksDB's internal statistics. > > Please have a look at it and let me know what you think. Since I am not a > RocksDB expert, I am thankful for any additional pair of eyes that > evaluates this KIP. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > Best regards, > Bruno >
[DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams
Hi all, this KIP describes the extension of the Kafka Streams' metrics to include RocksDB's internal statistics. Please have a look at it and let me know what you think. Since I am not a RocksDB expert, I am thankful for any additional pair of eyes that evaluates this KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams Best regards, Bruno
