Re: Why RocksDB metrics cache-usage is larger than cache-capacity

Sorry, it was probably an observation mistake. I export the metrics to Prometheus and query the result on grafana, actually the usage will not exceed the capacity Thanks, Lei On Fri, Apr 19, 2024 at 9:55 AM Hangxiang Yu wrote: > Hi, Lei. > It's indeed a bit confusing. Could you share the rela

Re: Why RocksDB metrics cache-usage is larger than cache-capacity

Hi, Lei. It's indeed a bit confusing. Could you share the related rocksdb log which may contain more detailed info ? On Fri, Apr 12, 2024 at 12:49 PM Lei Wang wrote: > > I enable RocksDB native metrics and do some performance tuning. > > state.backend.rocksdb.block.cache-size is set to 128m，4 sl

Why RocksDB metrics cache-usage is larger than cache-capacity

I enable RocksDB native metrics and do some performance tuning. state.backend.rocksdb.block.cache-size is set to 128m，4 slots for each TaskManager. The observed result for one specific parallel slot: state.backend.rocksdb.metrics.block-cache-capacity is about 14.5M state.backend.rocksdb.metric

Re: RocksDB metrics for effective memory consumption

usage [1]. Please note that all RocksDB instances within same slot would share the same block cache, they will report same usage. [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-metrics-block-cache-usage Best Yun

RocksDB metrics for effective memory consumption

Hi, I am working on the analysis of the memory consumption of RocksDB state backend for simple DAGs. I would like to check fine-grained memory utilization of RocksDB with the native metrics (reported on Prometheus+Grafana). RocksDB uses Managed memory allocated to each TaskManager but this valu

Re: Recommendation about RocksDB Metrics ?

December 9, 2020 12:11 To: Khachatryan Roman Cc: Truong Duc Kien ; Yun Tang ; user Subject: Re: Recommendation about RocksDB Metrics ? just a data point. we actually enabled all RocksDb metrics by default (including very large jobs in terms of parallelism and state size). We didn't see any signif

Re: Recommendation about RocksDB Metrics ?

just a data point. we actually enabled all RocksDb metrics by default (including very large jobs in terms of parallelism and state size). We didn't see any significant performance impact. There is probably a small impact. At least, it didn't jump out for our workload. On Tue, Dec 8, 20

Re: Recommendation about RocksDB Metrics ?

Hi Kien, I am pulling in Yun who might know better. Regards, Roman On Sun, Dec 6, 2020 at 3:52 AM Truong Duc Kien wrote: > Hi all, > > We are thinking about enabling RocksDB metrics to better monitor our > pipeline. However, since they will have performance impact, we will

Recommendation about RocksDB Metrics ?

Hi all, We are thinking about enabling RocksDB metrics to better monitor our pipeline. However, since they will have performance impact, we will have to be selective about which metrics we use. Does anyone have experience about which metrics are more important than the others ? And what metrics

Re: Rocksdb Metrics

- >> *From:* Stefan Richter >> *Sent:* Wednesday, September 26, 2018 0:56 >> *To:* Sayat Satybaldiyev >> *Cc:* user@flink.apache.org >> *Subject:* Re: Rocksdb Metrics >> >> Hi, >> >> this feature is tracked here >> https://is

Re: Rocksdb Metrics

7;s db folder. > > Best > Yun > -- > *From:* Stefan Richter > *Sent:* Wednesday, September 26, 2018 0:56 > *To:* Sayat Satybaldiyev > *Cc:* user@flink.apache.org > *Subject:* Re: Rocksdb Metrics > > Hi, > > this feature is tracked here > https://issues.apache

Re: Rocksdb Metrics

interval), and you could find the information LOG file under rocksDB state-backend's db folder. Best Yun From: Stefan Richter Sent: Wednesday, September 26, 2018 0:56 To: Sayat Satybaldiyev Cc: user@flink.apache.org Subject: Re: Rocksdb Metrics Hi, this featu

Re: Rocksdb Metrics

Hi, this feature is tracked here https://issues.apache.org/jira/browse/FLINK-10423 Best, Stefan > Am 25.09.2018 um 17:51 schrieb Sayat Satybaldiyev : > > Flink provides a rich number of metrics. However, I didn't find any metrics > for rocks

Rocksdb Metrics

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. Is there are any metrics for the rocksdb backend that Flink exposes?