It looks to me like all global metrics have to do with WAL. Why not call
them IgniteWalMetrics then?
On Tue, May 30, 2017 at 11:58 AM, Denis Magda <dma...@apache.org> wrote:
> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
>
> The issue is that we can’t combine all of them because these
>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >> public double getPageEvictionRate();
> >>
> >> public double getPageMissRate();
> >>
> >> public long getDirtyPages();
> >>
> >> public long getPagesOnDisk();
> >> }
>
>
> are calculated per memory policy (memory region) while these
>
> >> public interface PersistenceStoreMetrics {
> >> //WAL-related metrics
> >> public double getWalLoggingRate();
> >>
> >> public int getWalArchiveSegments();
> >>
> >> public double getWalFsyncAverage();
> >>
> >> //checkpoint-related metrics
> >> public double getCpTime();
> >>
> >> public double getCpFsyncTime();
> >>
> >> public long getCpTotalPages();
> >>
> >> public long[] getCpPagesByType();
> >>
> >> public long getCpCopiedOnWritePages();
> >> }
>
> are global.
>
>
> —
> Denis
>
> > On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <dsetrak...@apache.org>
> wrote:
> >
> > Several questions:
> >
> > 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should
> be
> > consistent with other time metrics, and should be called
> > "getWalFsyncAverageTime()"
> > 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
> > 3. Also, how about WAL total pages count or total size, whichever one
> is
> > easier?
> > 4. How about the WAL size that has not been checkpointed yet?
> > 5. How about WAL history size (the number of WAL files that we are
> > rotating through)?
> >
> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
> >
> > D.
> >
> >
> > On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> I have an idea of how we could clarify proposed above approach to
> >> collecting metrics for PDS.
> >>
> >> I suggest to split all metrics into two major groups (disk-related and
> >> memory-related) and group them in the following interfaces:
> >>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >> public double getPageEvictionRate();
> >>
> >> public double getPageMissRate();
> >>
> >> public long getDirtyPages();
> >>
> >> public long getPagesOnDisk();
> >> }
> >>
> >> public interface PersistenceStoreMetrics {
> >> //WAL-related metrics
> >> public double getWalLoggingRate();
> >>
> >> public int getWalArchiveSegments();
> >>
> >> public double getWalFsyncAverage();
> >>
> >> //checkpoint-related metrics
> >> public double getCpTime();
> >>
> >> public double getCpFsyncTime();
> >>
> >> public long getCpTotalPages();
> >>
> >> public long[] getCpPagesByType();
> >>
> >> public long getCpCopiedOnWritePages();
> >> }
> >>
> >> From my standpoint it is important to distinguish between PDS metrics
> that
> >> are related mostly to memory management (from the first interface) and
> >> metrics bound to disk operations (from the second interface).
> >>
> >> First group of metrics is useful when collected per Memory Policy. It
> >> doesn't provide a lot of information when collected across many Memory
> >> Policies.
> >>
> >> Second group is now global (this may change in the future if Persistence
> >> Policies are introduced) and allows to easily spot any high-level issues
> >> with durable storage configuration or performance. More specific
> questions
> >> like "how this Memory Policy loads a durable storage subsystem?" may be
> >> deduced from PersistenceMemoryMetrics indicators.
> >>
> >> Denis, does it make sense for you?
> >>
> >> Any other ideas/questions/improvements or metrics to add to these
> >> interfaces?
> >>
> >> I would like to discuss all the details here before creating any tickets
> >> for the change.
> >>
> >> Thanks,
> >> Sergey.
> >>
> >>
> >> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dma...@apache.org> wrote:
> >>
> >>> Dmitriy,
> >>>
> >>> I have some high level concerns.
> >>>
> >>> Presently, Ignite already exposes memory/data related metrics via
> >>> MemoryMetrics and CacheMetrics interface.
> >>>
> >>> Looking at PersistentStoreMetrics interface I can’t get why methods
> like
> >>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are
> NOT
> >>> in MemoryMetrics interface.
> >>>
> >>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
> >>> contains a big subset of the methods from PersistentStoreMetric. There
> >> just
> >>> should be some API call that either returns global metrics or cache
> >>> specific ones. Plus, some of the methods might be added to existing
> >>> CacheInterface. Why not.
> >>>
> >>> I prefer to address this first. Guys, any other thoughts on this?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
> >>> dmitriy.govoruk...@gmail.com> wrote:
> >>>>
> >>>> Folk,
> >>>>
> >>>> As you know, ignite 2.1 will contain new module (pds), it will be
> >>>> provide ability to store data on disk. Let's discuss what type of
> >>>> metrics we need for this?
> >>>> I think it must be metrics per memory policy, per cache, checkpoint,
> >>>> and global metrics which will be aggregate all metrics.
> >>>>
> >>>> I did sketch.
> >>>>
> >>>> PersistentStoreMetrics.java
> >>>>
> >>>> public interface PersistentStoreMetrics {
> >>>>
> >>>> // Global metrics.
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long walLog();
> >>>>
> >>>> public long walLogSize();
> >>>>
> >>>> // Frequency.
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWrite();
> >>>>
> >>>> public long getFsync();
> >>>>
> >>>> public long getWal();
> >>>>
> >>>> public long getAverageWalFsyncTime();
> >>>>
> >>>> // Per cache.
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(String name);
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(int cacheId);
> >>>>
> >>>> // For last checkpoint.
> >>>>
> >>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCacheMetrics.java
> >>>>
> >>>> public interface PersistentStoreCacheMetrics {
> >>>>
> >>>> public String name();
> >>>>
> >>>> public double getFillFactor();
> >>>>
> >>>> public double getFillFactor(int part);
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWritten();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCheckpointMetrics.java
> >>>>
> >>>> public interface PersistentStoreCheckpointMetrics {
> >>>>
> >>>> public long getTotalPages();
> >>>>
> >>>> //TODO Page type is internal?
> >>>> public long[] pagesType();
> >>>>
> >>>> public long getExecutingTime();
> >>>>
> >>>> public long getFsyncTime();
> >>>>
> >>>> public long getPagesCopyOnWrite();
> >>>> }
> >>>
> >>>
> >>
>
>
