Thanks. At least for us it doesn't matter how exact the number is. I would expect most users to be only interested in monitoring if the total state size keeps growing (rapidly), or remains about the same. I suppose all of the options that you suggested would satisfy this need?
On Fri, Apr 20, 2018 at 12:53 PM, Stefan Richter < s.rich...@data-artisans.com> wrote: > Hi, > > for incremental checkpoints, it is only showing the size of the deltas. It > would probably also be possible to report the full size, but the current > reporting and UI is only supporting to deliver a single value. In general, > some things are rather hard to report. For example, for the heap based > backend, is the state size the size of the serialized data or the size of > the heap objects? > Another remark about key count: the key count is easy to determine for the > heap based backend, but there is no (efficient) method in RocksDb that > gives the key count (because of the way RocksDB works). In this case, > afaik, we have the (inefficient) option to iterate all keys and count or > use the (efficient) estimated key count is supported by RocksDB. > > Best, > Stefan > > > Am 04.01.2018 um 19:23 schrieb Steven Wu <stevenz...@gmail.com>: > > Aljoscha/Stefan, > > if incremental checkpoint is enabled, I assume the "checkpoint size" is > only the delta/incremental size (not the full state size), right? > > Thanks, > Steven > > > On Thu, Jan 4, 2018 at 5:18 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > >> Hi, >> >> I'm afraid there is currently no metrics around state. I see that it's >> very good to have so I'm putting it on my list of stuff that we should have >> at some point. >> >> One thing that comes to mind is checking the size of checkpoints, which >> gives you an indirect way of figuring out how big state is but that's not >> very exact, i.e. doesn't give you "number of keys" or some such. >> >> Best, >> Aljoscha >> >> > On 20. Dec 2017, at 08:09, Netzer, Liron <liron.net...@citi.com> wrote: >> > >> > Ufuk, Thanks for replying ! >> > >> > Aljoscha, can you please assist with the questions below? >> > >> > Thanks, >> > Liron >> > >> > -----Original Message----- >> > From: Ufuk Celebi [mailto:u...@apache.org] >> > Sent: Friday, December 15, 2017 3:06 PM >> > To: Netzer, Liron [ICG-IT] >> > Cc: user@flink.apache.org >> > Subject: Re: Flink State monitoring >> > >> > Hey Liron, >> > >> > unfortunately, there are no built-in metrics related to state. In >> general, exposing the actual values as metrics is problematic, but exposing >> summary statistics would be a good idea. I'm not aware of a good work >> around at the moment that would work in the general case (taking into >> account state restore, etc.). >> > >> > Let me pull in Aljoscha (cc'd) who knows the state backend internals >> well. >> > >> > @Aljoscha: >> > 1) Are there any plans to expose keyed state related metrics (like >> number of keys)? >> > 2) Is there a way to work around the lack of these metrics in 1.3? >> > >> > – Ufuk >> > >> > On Thu, Dec 14, 2017 at 10:55 AM, Netzer, Liron <liron.net...@citi.com> >> wrote: >> >> Hi group, >> >> >> >> >> >> >> >> We are using Flink keyed state in several operators. >> >> >> >> Is there an easy was to expose the data that is stored in the state, >> i.e. >> >> the key and the values? >> >> >> >> This is needed for both monitoring as well as debugging. We would like >> >> to understand how many key+values are stored in each state and also to >> >> view the data itself. >> >> >> >> I know that there is the "Queryable state" option, but this is still >> >> in Beta, and doesn't really give us what we want easily. >> >> >> >> >> >> >> >> >> >> >> >> *We are using Flink 1.3.2 with Java. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Liron >> >> > >
