> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.
The issue is that we can’t combine all of them because these
>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>> public double getPageEvictionRate();
>>
>> public double getPageMissRate();
>>
>> public long getDirtyPages();
>>
>> public long getPagesOnDisk();
>> }
are calculated per memory policy (memory region) while these
>> public interface PersistenceStoreMetrics {
>> //WAL-related metrics
>> public double getWalLoggingRate();
>>
>> public int getWalArchiveSegments();
>>
>> public double getWalFsyncAverage();
>>
>> //checkpoint-related metrics
>> public double getCpTime();
>>
>> public double getCpFsyncTime();
>>
>> public long getCpTotalPages();
>>
>> public long[] getCpPagesByType();
>>
>> public long getCpCopiedOnWritePages();
>> }
are global.
—
Denis
> On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <dsetrak...@apache.org> wrote:
>
> Several questions:
>
> 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should be
> consistent with other time metrics, and should be called
> "getWalFsyncAverageTime()"
> 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
> 3. Also, how about WAL total pages count or total size, whichever one is
> easier?
> 4. How about the WAL size that has not been checkpointed yet?
> 5. How about WAL history size (the number of WAL files that we are
> rotating through)?
>
> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.
>
> D.
>
>
> On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <sergey.chugu...@gmail.com>
> wrote:
>
>> Hello Denis,
>>
>> I have an idea of how we could clarify proposed above approach to
>> collecting metrics for PDS.
>>
>> I suggest to split all metrics into two major groups (disk-related and
>> memory-related) and group them in the following interfaces:
>>
>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>> public double getPageEvictionRate();
>>
>> public double getPageMissRate();
>>
>> public long getDirtyPages();
>>
>> public long getPagesOnDisk();
>> }
>>
>> public interface PersistenceStoreMetrics {
>> //WAL-related metrics
>> public double getWalLoggingRate();
>>
>> public int getWalArchiveSegments();
>>
>> public double getWalFsyncAverage();
>>
>> //checkpoint-related metrics
>> public double getCpTime();
>>
>> public double getCpFsyncTime();
>>
>> public long getCpTotalPages();
>>
>> public long[] getCpPagesByType();
>>
>> public long getCpCopiedOnWritePages();
>> }
>>
>> From my standpoint it is important to distinguish between PDS metrics that
>> are related mostly to memory management (from the first interface) and
>> metrics bound to disk operations (from the second interface).
>>
>> First group of metrics is useful when collected per Memory Policy. It
>> doesn't provide a lot of information when collected across many Memory
>> Policies.
>>
>> Second group is now global (this may change in the future if Persistence
>> Policies are introduced) and allows to easily spot any high-level issues
>> with durable storage configuration or performance. More specific questions
>> like "how this Memory Policy loads a durable storage subsystem?" may be
>> deduced from PersistenceMemoryMetrics indicators.
>>
>> Denis, does it make sense for you?
>>
>> Any other ideas/questions/improvements or metrics to add to these
>> interfaces?
>>
>> I would like to discuss all the details here before creating any tickets
>> for the change.
>>
>> Thanks,
>> Sergey.
>>
>>
>> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dma...@apache.org> wrote:
>>
>>> Dmitriy,
>>>
>>> I have some high level concerns.
>>>
>>> Presently, Ignite already exposes memory/data related metrics via
>>> MemoryMetrics and CacheMetrics interface.
>>>
>>> Looking at PersistentStoreMetrics interface I can’t get why methods like
>>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
>>> in MemoryMetrics interface.
>>>
>>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>>> contains a big subset of the methods from PersistentStoreMetric. There
>> just
>>> should be some API call that either returns global metrics or cache
>>> specific ones. Plus, some of the methods might be added to existing
>>> CacheInterface. Why not.
>>>
>>> I prefer to address this first. Guys, any other thoughts on this?
>>>
>>> —
>>> Denis
>>>
>>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>>> dmitriy.govoruk...@gmail.com> wrote:
>>>>
>>>> Folk,
>>>>
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>>
>>>> I did sketch.
>>>>
>>>> PersistentStoreMetrics.java
>>>>
>>>> public interface PersistentStoreMetrics {
>>>>
>>>> // Global metrics.
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long walLog();
>>>>
>>>> public long walLogSize();
>>>>
>>>> // Frequency.
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWrite();
>>>>
>>>> public long getFsync();
>>>>
>>>> public long getWal();
>>>>
>>>> public long getAverageWalFsyncTime();
>>>>
>>>> // Per cache.
>>>>
>>>> public PersistentStoreCacheMetrics cache(String name);
>>>>
>>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>>
>>>> // For last checkpoint.
>>>>
>>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCacheMetrics.java
>>>>
>>>> public interface PersistentStoreCacheMetrics {
>>>>
>>>> public String name();
>>>>
>>>> public double getFillFactor();
>>>>
>>>> public double getFillFactor(int part);
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWritten();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCheckpointMetrics.java
>>>>
>>>> public interface PersistentStoreCheckpointMetrics {
>>>>
>>>> public long getTotalPages();
>>>>
>>>> //TODO Page type is internal?
>>>> public long[] pagesType();
>>>>
>>>> public long getExecutingTime();
>>>>
>>>> public long getFsyncTime();
>>>>
>>>> public long getPagesCopyOnWrite();
>>>> }
>>>
>>>
>>
