I’m afraid that in the future we might need to add non WAL related metrics
under this interface. This is why it might make sense to keep the name generic.
—
Denis
> On May 30, 2017, at 12:29 PM, Dmitriy Setrakyan <dsetrak...@apache.org> wrote:
>
> It looks to me like all global metrics have to do with WAL. Why not call
> them IgniteWalMetrics then?
>
> On Tue, May 30, 2017 at 11:58 AM, Denis Magda <dma...@apache.org> wrote:
>
>>> Also, I see your argument about separating memory from disk metrics, but
>> I
>>> think it makes sense to combine them, because it is hard to separate them
>>> from user standpoint. Let's call them IgniteStorageMetrics or something
>> of
>>> the kind.
>>
>> The issue is that we can’t combine all of them because these
>>
>>>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>>> public double getPageEvictionRate();
>>>>
>>>> public double getPageMissRate();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getPagesOnDisk();
>>>> }
>>
>>
>> are calculated per memory policy (memory region) while these
>>
>>>> public interface PersistenceStoreMetrics {
>>>> //WAL-related metrics
>>>> public double getWalLoggingRate();
>>>>
>>>> public int getWalArchiveSegments();
>>>>
>>>> public double getWalFsyncAverage();
>>>>
>>>> //checkpoint-related metrics
>>>> public double getCpTime();
>>>>
>>>> public double getCpFsyncTime();
>>>>
>>>> public long getCpTotalPages();
>>>>
>>>> public long[] getCpPagesByType();
>>>>
>>>> public long getCpCopiedOnWritePages();
>>>> }
>>
>> are global.
>>
>>
>> —
>> Denis
>>
>>> On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <dsetrak...@apache.org>
>> wrote:
>>>
>>> Several questions:
>>>
>>> 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should
>> be
>>> consistent with other time metrics, and should be called
>>> "getWalFsyncAverageTime()"
>>> 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
>>> 3. Also, how about WAL total pages count or total size, whichever one
>> is
>>> easier?
>>> 4. How about the WAL size that has not been checkpointed yet?
>>> 5. How about WAL history size (the number of WAL files that we are
>>> rotating through)?
>>>
>>> Also, I see your argument about separating memory from disk metrics, but
>> I
>>> think it makes sense to combine them, because it is hard to separate them
>>> from user standpoint. Let's call them IgniteStorageMetrics or something
>> of
>>> the kind.
>>>
>>> D.
>>>
>>>
>>> On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com>
>>> wrote:
>>>
>>>> Hello Denis,
>>>>
>>>> I have an idea of how we could clarify proposed above approach to
>>>> collecting metrics for PDS.
>>>>
>>>> I suggest to split all metrics into two major groups (disk-related and
>>>> memory-related) and group them in the following interfaces:
>>>>
>>>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>>> public double getPageEvictionRate();
>>>>
>>>> public double getPageMissRate();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getPagesOnDisk();
>>>> }
>>>>
>>>> public interface PersistenceStoreMetrics {
>>>> //WAL-related metrics
>>>> public double getWalLoggingRate();
>>>>
>>>> public int getWalArchiveSegments();
>>>>
>>>> public double getWalFsyncAverage();
>>>>
>>>> //checkpoint-related metrics
>>>> public double getCpTime();
>>>>
>>>> public double getCpFsyncTime();
>>>>
>>>> public long getCpTotalPages();
>>>>
>>>> public long[] getCpPagesByType();
>>>>
>>>> public long getCpCopiedOnWritePages();
>>>> }
>>>>
>>>> From my standpoint it is important to distinguish between PDS metrics
>> that
>>>> are related mostly to memory management (from the first interface) and
>>>> metrics bound to disk operations (from the second interface).
>>>>
>>>> First group of metrics is useful when collected per Memory Policy. It
>>>> doesn't provide a lot of information when collected across many Memory
>>>> Policies.
>>>>
>>>> Second group is now global (this may change in the future if Persistence
>>>> Policies are introduced) and allows to easily spot any high-level issues
>>>> with durable storage configuration or performance. More specific
>> questions
>>>> like "how this Memory Policy loads a durable storage subsystem?" may be
>>>> deduced from PersistenceMemoryMetrics indicators.
>>>>
>>>> Denis, does it make sense for you?
>>>>
>>>> Any other ideas/questions/improvements or metrics to add to these
>>>> interfaces?
>>>>
>>>> I would like to discuss all the details here before creating any tickets
>>>> for the change.
>>>>
>>>> Thanks,
>>>> Sergey.
>>>>
>>>>
>>>> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dma...@apache.org> wrote:
>>>>
>>>>> Dmitriy,
>>>>>
>>>>> I have some high level concerns.
>>>>>
>>>>> Presently, Ignite already exposes memory/data related metrics via
>>>>> MemoryMetrics and CacheMetrics interface.
>>>>>
>>>>> Looking at PersistentStoreMetrics interface I can’t get why methods
>> like
>>>>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are
>> NOT
>>>>> in MemoryMetrics interface.
>>>>>
>>>>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>>>>> contains a big subset of the methods from PersistentStoreMetric. There
>>>> just
>>>>> should be some API call that either returns global metrics or cache
>>>>> specific ones. Plus, some of the methods might be added to existing
>>>>> CacheInterface. Why not.
>>>>>
>>>>> I prefer to address this first. Guys, any other thoughts on this?
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>>>>> dmitriy.govoruk...@gmail.com> wrote:
>>>>>>
>>>>>> Folk,
>>>>>>
>>>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>>>> provide ability to store data on disk. Let's discuss what type of
>>>>>> metrics we need for this?
>>>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>>>> and global metrics which will be aggregate all metrics.
>>>>>>
>>>>>> I did sketch.
>>>>>>
>>>>>> PersistentStoreMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreMetrics {
>>>>>>
>>>>>> // Global metrics.
>>>>>>
>>>>>> public long getMemorySize();
>>>>>>
>>>>>> public long getDiskSize();
>>>>>>
>>>>>> public long getPagesInMemory();
>>>>>>
>>>>>> public long getPagesSizeInMemory();
>>>>>>
>>>>>> public long getPagesOnDisk();
>>>>>>
>>>>>> public long getPagesSizeOnDisk();
>>>>>>
>>>>>> public long getFreePages();
>>>>>>
>>>>>> public long getFreePagesSize();
>>>>>>
>>>>>> public long getDirtyPages();
>>>>>>
>>>>>> public long getDirtyPagesSize();
>>>>>>
>>>>>> public long walLog();
>>>>>>
>>>>>> public long walLogSize();
>>>>>>
>>>>>> // Frequency.
>>>>>>
>>>>>> public long getPagesRead();
>>>>>>
>>>>>> public long getPagesWrite();
>>>>>>
>>>>>> public long getFsync();
>>>>>>
>>>>>> public long getWal();
>>>>>>
>>>>>> public long getAverageWalFsyncTime();
>>>>>>
>>>>>> // Per cache.
>>>>>>
>>>>>> public PersistentStoreCacheMetrics cache(String name);
>>>>>>
>>>>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>>>>
>>>>>> // For last checkpoint.
>>>>>>
>>>>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>>>> }
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>> PersistentStoreCacheMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreCacheMetrics {
>>>>>>
>>>>>> public String name();
>>>>>>
>>>>>> public double getFillFactor();
>>>>>>
>>>>>> public double getFillFactor(int part);
>>>>>>
>>>>>> public long getMemorySize();
>>>>>>
>>>>>> public long getDiskSize();
>>>>>>
>>>>>> public long getPagesInMemory();
>>>>>>
>>>>>> public long getPagesSizeInMemory();
>>>>>>
>>>>>> public long getPagesOnDisk();
>>>>>>
>>>>>> public long getPagesSizeOnDisk();
>>>>>>
>>>>>> public long getFreePages();
>>>>>>
>>>>>> public long getFreePagesSize();
>>>>>>
>>>>>> public long getDirtyPages();
>>>>>>
>>>>>> public long getDirtyPagesSize();
>>>>>>
>>>>>> public long getPagesRead();
>>>>>>
>>>>>> public long getPagesWritten();
>>>>>> }
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>> PersistentStoreCheckpointMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreCheckpointMetrics {
>>>>>>
>>>>>> public long getTotalPages();
>>>>>>
>>>>>> //TODO Page type is internal?
>>>>>> public long[] pagesType();
>>>>>>
>>>>>> public long getExecutingTime();
>>>>>>
>>>>>> public long getFsyncTime();
>>>>>>
>>>>>> public long getPagesCopyOnWrite();
>>>>>> }
>>>>>
>>>>>
>>>>
>>
>>
