Re: Metrics for MVCC caches
Hi. There is another similar case, what if update actually change nothing? BEGIN; INSERT INTO person(id, name) VALUES(1, 'ivan'); UPDATE person SET name = 'ivan' WHERE id = 1; COMMIT; Should it we take into account both operations or not? I'd think current implementation is correct one as TX will make only one update in both cases. On Fri, Aug 17, 2018 at 4:54 PM Павлухин Иван wrote: > Hi team, > > I need you opinion again. I wrote some tests for metrics and felt a > confusion with metrics count for the same key update during transaction. > Imagine following case: > > BEGIN; > INSERT INTO person(id, name) VALUES(1, 'ivan'); > UPDATE person SET name = 'vanya' WHERE id = 1; > COMMIT; > > My intuition said me that puts count should be increased by 2 for such > case. But current (non-mvcc) implementation increments by 1. I am not sure > that it is proper semantics. On the other hand, I think that it is better > to preserve existing behavior instead of making exception for mvcc. It will > cause even more confusion. > > So, if nobody has objections I will go with existing semantics. > -- Best regards, Andrey V. Mashenkov
Re: Metrics for MVCC caches
Hi team, I need you opinion again. I wrote some tests for metrics and felt a confusion with metrics count for the same key update during transaction. Imagine following case: BEGIN; INSERT INTO person(id, name) VALUES(1, 'ivan'); UPDATE person SET name = 'vanya' WHERE id = 1; COMMIT; My intuition said me that puts count should be increased by 2 for such case. But current (non-mvcc) implementation increments by 1. I am not sure that it is proper semantics. On the other hand, I think that it is better to preserve existing behavior instead of making exception for mvcc. It will cause even more confusion. So, if nobody has objections I will go with existing semantics.
Re: Metrics for MVCC caches
Hi Alex,
Thank you for referring spec. It really gives strict point how JCache
metrics should be counted.
So, I will start implementation of metrics for MVCC caches according to
spec and existing behavior.
As for improper "removals" calculation, I filed a ticket
https://issues.apache.org/jira/browse/IGNITE-9300
2018-08-10 17:48 GMT+03:00 Alex Plehanov :
> Hi, Ivan
>
> Looks like a bug. Metrics counting described at JSR107 (JCache)
> specification. It says, that "cache removals" metric should be incremented
> if remove() method returns true. Since there are no matching keys in the
> first remove() call should be found, remove() should return false and
> "cache removals" metric should not be affected.
>
> 2018-08-10 12:56 GMT+03:00 Павлухин Иван :
>
> > Hi,
> >
> > Dmitriy thanks for note, I will keep metrics IEP in mind.
> >
> > Roman it sounds quite a straightforward approach. Moreover current cache
> > transactions follow it: there could be a number of invisible actions in
> > private workspace (e.g. creating and then deleting the same entry) but
> only
> > final changes are metered.
> >
> > Also, I observed strange behavior for TRANSACTIONAL cache operations. For
> > following case metrics surprised me (numbers in comments):
> >
> > public void testPutRemove() throws Exception {
> > cache.remove(1);
> > System.out.println(cache.metrics().getCacheRemovals()); // 1
> >
> > cache.put(1, "a");
> >
> > cache.remove(1);
> > System.out.println(cache.metrics().getCacheRemovals()); // 2
> > }
> >
> > While for ATOMIC cache I see 0 and 1 which looks much better. And similar
> > effect is observed for "replace" operations.
> >
> > Is it a bug? Do we have some document describing how metrics should be
> > counted?
> >
> > 2018-08-08 21:11 GMT+03:00 Dmitriy Setrakyan :
> >
> > > I think we should be updating the IEP with metrics specifications in
> > > parallel.
> > >
> > > D.
> > >
> > > On Wed, Aug 8, 2018, 05:32 Roman Kondakov
> > > wrote:
> > >
> > > > Hi Ivan!
> > > >
> > > > In my opinion we need to preserve the essence of the metrics: if we
> > > > didn't consider dirty writes as updates before MVCC implementation,
> we
> > > > also shouldn't count these writes in MVCC metrics implementation too.
> > > > So, I think we need to count only committed entries. But we can count
> > > > this dirty writes as additional metrics.
> > > >
> > > > Also additional metrics concerning MVCC could be:
> > > >
> > > > - Average count of the active transactions per snapshot
> > > >
> > > > - Average quantity of versions per key
> > > >
> > > >
> > > > --
> > > >
> > > > Roman Kondakov
> > > >
> > > >
> > > > On 07.08.2018 17:17, Павлухин Иван wrote:
> > > > > Hi Igniters,
> > > > >
> > > > > I am working on cache metrics support for caches with enabled MVCC.
> > As
> > > > you
> > > > > may know, during MVCC transaction execution entry versions are
> > written
> > > to
> > > > > storage right away (without deferring until commit). So, it is not
> > > > obvious
> > > > > for me if we should update writes count right away or defer it
> until
> > > > > commit. On one hand, MVCC entry version write is not "true" write
> > until
> > > > > commit. On the other hand, it definitely changes the storage. How
> do
> > we
> > > > > interpret write metrics in Ignite philosophy?
> > > > >
> > > > > Also, it seems that bunch of MVCC specific metrics could be
> > > introduced. I
> > > > > would like to collect some thoughts about it. Which such metrics
> come
> > > to
> > > > > your mind?
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>
--
Best regards,
Ivan Pavlukhin
Re: Metrics for MVCC caches
Hi, Ivan
Looks like a bug. Metrics counting described at JSR107 (JCache)
specification. It says, that "cache removals" metric should be incremented
if remove() method returns true. Since there are no matching keys in the
first remove() call should be found, remove() should return false and
"cache removals" metric should not be affected.
2018-08-10 12:56 GMT+03:00 Павлухин Иван :
> Hi,
>
> Dmitriy thanks for note, I will keep metrics IEP in mind.
>
> Roman it sounds quite a straightforward approach. Moreover current cache
> transactions follow it: there could be a number of invisible actions in
> private workspace (e.g. creating and then deleting the same entry) but only
> final changes are metered.
>
> Also, I observed strange behavior for TRANSACTIONAL cache operations. For
> following case metrics surprised me (numbers in comments):
>
> public void testPutRemove() throws Exception {
> cache.remove(1);
> System.out.println(cache.metrics().getCacheRemovals()); // 1
>
> cache.put(1, "a");
>
> cache.remove(1);
> System.out.println(cache.metrics().getCacheRemovals()); // 2
> }
>
> While for ATOMIC cache I see 0 and 1 which looks much better. And similar
> effect is observed for "replace" operations.
>
> Is it a bug? Do we have some document describing how metrics should be
> counted?
>
> 2018-08-08 21:11 GMT+03:00 Dmitriy Setrakyan :
>
> > I think we should be updating the IEP with metrics specifications in
> > parallel.
> >
> > D.
> >
> > On Wed, Aug 8, 2018, 05:32 Roman Kondakov
> > wrote:
> >
> > > Hi Ivan!
> > >
> > > In my opinion we need to preserve the essence of the metrics: if we
> > > didn't consider dirty writes as updates before MVCC implementation, we
> > > also shouldn't count these writes in MVCC metrics implementation too.
> > > So, I think we need to count only committed entries. But we can count
> > > this dirty writes as additional metrics.
> > >
> > > Also additional metrics concerning MVCC could be:
> > >
> > > - Average count of the active transactions per snapshot
> > >
> > > - Average quantity of versions per key
> > >
> > >
> > > --
> > >
> > > Roman Kondakov
> > >
> > >
> > > On 07.08.2018 17:17, Павлухин Иван wrote:
> > > > Hi Igniters,
> > > >
> > > > I am working on cache metrics support for caches with enabled MVCC.
> As
> > > you
> > > > may know, during MVCC transaction execution entry versions are
> written
> > to
> > > > storage right away (without deferring until commit). So, it is not
> > > obvious
> > > > for me if we should update writes count right away or defer it until
> > > > commit. On one hand, MVCC entry version write is not "true" write
> until
> > > > commit. On the other hand, it definitely changes the storage. How do
> we
> > > > interpret write metrics in Ignite philosophy?
> > > >
> > > > Also, it seems that bunch of MVCC specific metrics could be
> > introduced. I
> > > > would like to collect some thoughts about it. Which such metrics come
> > to
> > > > your mind?
> > > >
> > >
> > >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Metrics for MVCC caches
Hi,
Dmitriy thanks for note, I will keep metrics IEP in mind.
Roman it sounds quite a straightforward approach. Moreover current cache
transactions follow it: there could be a number of invisible actions in
private workspace (e.g. creating and then deleting the same entry) but only
final changes are metered.
Also, I observed strange behavior for TRANSACTIONAL cache operations. For
following case metrics surprised me (numbers in comments):
public void testPutRemove() throws Exception {
cache.remove(1);
System.out.println(cache.metrics().getCacheRemovals()); // 1
cache.put(1, "a");
cache.remove(1);
System.out.println(cache.metrics().getCacheRemovals()); // 2
}
While for ATOMIC cache I see 0 and 1 which looks much better. And similar
effect is observed for "replace" operations.
Is it a bug? Do we have some document describing how metrics should be
counted?
2018-08-08 21:11 GMT+03:00 Dmitriy Setrakyan :
> I think we should be updating the IEP with metrics specifications in
> parallel.
>
> D.
>
> On Wed, Aug 8, 2018, 05:32 Roman Kondakov
> wrote:
>
> > Hi Ivan!
> >
> > In my opinion we need to preserve the essence of the metrics: if we
> > didn't consider dirty writes as updates before MVCC implementation, we
> > also shouldn't count these writes in MVCC metrics implementation too.
> > So, I think we need to count only committed entries. But we can count
> > this dirty writes as additional metrics.
> >
> > Also additional metrics concerning MVCC could be:
> >
> > - Average count of the active transactions per snapshot
> >
> > - Average quantity of versions per key
> >
> >
> > --
> >
> > Roman Kondakov
> >
> >
> > On 07.08.2018 17:17, Павлухин Иван wrote:
> > > Hi Igniters,
> > >
> > > I am working on cache metrics support for caches with enabled MVCC. As
> > you
> > > may know, during MVCC transaction execution entry versions are written
> to
> > > storage right away (without deferring until commit). So, it is not
> > obvious
> > > for me if we should update writes count right away or defer it until
> > > commit. On one hand, MVCC entry version write is not "true" write until
> > > commit. On the other hand, it definitely changes the storage. How do we
> > > interpret write metrics in Ignite philosophy?
> > >
> > > Also, it seems that bunch of MVCC specific metrics could be
> introduced. I
> > > would like to collect some thoughts about it. Which such metrics come
> to
> > > your mind?
> > >
> >
> >
>
--
Best regards,
Ivan Pavlukhin
Re: Metrics for MVCC caches
I think we should be updating the IEP with metrics specifications in parallel. D. On Wed, Aug 8, 2018, 05:32 Roman Kondakov wrote: > Hi Ivan! > > In my opinion we need to preserve the essence of the metrics: if we > didn't consider dirty writes as updates before MVCC implementation, we > also shouldn't count these writes in MVCC metrics implementation too. > So, I think we need to count only committed entries. But we can count > this dirty writes as additional metrics. > > Also additional metrics concerning MVCC could be: > > - Average count of the active transactions per snapshot > > - Average quantity of versions per key > > > -- > > Roman Kondakov > > > On 07.08.2018 17:17, Павлухин Иван wrote: > > Hi Igniters, > > > > I am working on cache metrics support for caches with enabled MVCC. As > you > > may know, during MVCC transaction execution entry versions are written to > > storage right away (without deferring until commit). So, it is not > obvious > > for me if we should update writes count right away or defer it until > > commit. On one hand, MVCC entry version write is not "true" write until > > commit. On the other hand, it definitely changes the storage. How do we > > interpret write metrics in Ignite philosophy? > > > > Also, it seems that bunch of MVCC specific metrics could be introduced. I > > would like to collect some thoughts about it. Which such metrics come to > > your mind? > > > >
Re: Metrics for MVCC caches
Hi Ivan! In my opinion we need to preserve the essence of the metrics: if we didn't consider dirty writes as updates before MVCC implementation, we also shouldn't count these writes in MVCC metrics implementation too. So, I think we need to count only committed entries. But we can count this dirty writes as additional metrics. Also additional metrics concerning MVCC could be: - Average count of the active transactions per snapshot - Average quantity of versions per key -- Roman Kondakov On 07.08.2018 17:17, Павлухин Иван wrote: Hi Igniters, I am working on cache metrics support for caches with enabled MVCC. As you may know, during MVCC transaction execution entry versions are written to storage right away (without deferring until commit). So, it is not obvious for me if we should update writes count right away or defer it until commit. On one hand, MVCC entry version write is not "true" write until commit. On the other hand, it definitely changes the storage. How do we interpret write metrics in Ignite philosophy? Also, it seems that bunch of MVCC specific metrics could be introduced. I would like to collect some thoughts about it. Which such metrics come to your mind?
Re: Metrics for MVCC caches
Dmitry, As with MVCC we can have multiple versions of each tuple at the same time it introduces storage overhead. It would be good to have a measure for such overhead. Another thing is related to VACUUM procedure. For protecting system from crash in case of high load it would be great to detect that VACUUM cleanup cannot keep up with new tuple versions creation rate. Currently it is just raw thoughts, I continue learning Ignite metrics framework. 2018-08-08 1:12 GMT+03:00 Dmitriy Setrakyan : > Ivan, > > To your 2nd question, can you suggest a few MVCC metrics to get the > discussion started? > > D. > > On Tue, Aug 7, 2018 at 9:17 AM, Павлухин Иван wrote: > > > Hi Igniters, > > > > I am working on cache metrics support for caches with enabled MVCC. As > you > > may know, during MVCC transaction execution entry versions are written to > > storage right away (without deferring until commit). So, it is not > obvious > > for me if we should update writes count right away or defer it until > > commit. On one hand, MVCC entry version write is not "true" write until > > commit. On the other hand, it definitely changes the storage. How do we > > interpret write metrics in Ignite philosophy? > > > > Also, it seems that bunch of MVCC specific metrics could be introduced. I > > would like to collect some thoughts about it. Which such metrics come to > > your mind? > > > > -- > > Best regards, > > Ivan Pavlukhin > > > -- Best regards, Ivan Pavlukhin
Re: Metrics for MVCC caches
Ivan, To your 2nd question, can you suggest a few MVCC metrics to get the discussion started? D. On Tue, Aug 7, 2018 at 9:17 AM, Павлухин Иван wrote: > Hi Igniters, > > I am working on cache metrics support for caches with enabled MVCC. As you > may know, during MVCC transaction execution entry versions are written to > storage right away (without deferring until commit). So, it is not obvious > for me if we should update writes count right away or defer it until > commit. On one hand, MVCC entry version write is not "true" write until > commit. On the other hand, it definitely changes the storage. How do we > interpret write metrics in Ignite philosophy? > > Also, it seems that bunch of MVCC specific metrics could be introduced. I > would like to collect some thoughts about it. Which such metrics come to > your mind? > > -- > Best regards, > Ivan Pavlukhin >
