[jira] [Commented] (IGNITE-7015) SQL: Index should be updated only when relevant values changed

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512275#comment-16512275
 ] 

Vladimir Ozerov commented on IGNITE-7015:
-

[~npordash], this specific fix is not likely to be merged in the nearest time. 
In-place updates are very complicated things and are not possible in general 
case. However, at this point we understand that slow index updates are caused 
by inefficient index implementation, and we are working on several tickets to 
improve it: 

https://issues.apache.org/jira/browse/IGNITE-8384

https://issues.apache.org/jira/browse/IGNITE-8386

 

> SQL: Index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>Priority: Major
>  Labels: iep-19, performance
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7015) SQL: Index should be updated only when relevant values changed

[Nick Pordash (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504989#comment-16504989
 ] 

Nick Pordash commented on IGNITE-7015:
--

[~vozerov] do you have a rough ETA when this might be released? The performance 
implications are so critical that I'm considering manually applying the PR to 
each release and rolling a custom build until then, which is obviously not a 
great situation to be in. For context, without this optimization in place I 
would need to have a cluster 3x-4x bigger just to absorb the excessive B+Tree 
updates.

> SQL: Index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>Priority: Major
>  Labels: iep-19, performance
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7015) SQL: index should be updated only when relevant values changed

[Roman Kondakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294317#comment-16294317
 ] 

Roman Kondakov commented on IGNITE-7015:


[~vozerov], I optimized my patch as you wrote in a comment above. A set of the 
indexed columns was added to {{GridH2Table}}. This set is changed on each index 
add/remove action. Temporary indexes are considered as normal indexes in this 
case. All comparison logic moved to {{GridH2Table}}.

As for {{GridQueryProcessor.store}} invocation - I went through the code 
execution in case of in-place update is possible and made sure that 
{{GridQueryProcessor.store}} is not executed in this case because in the method 
 {{GridCacheMapEntry.AtomicCacheUpdateClosure#update}} we can see this snippet:

{code:java}
treeOp = oldRow != null && oldRow.link() == newRow.link() ?
IgniteTree.OperationType.NOOP : 
IgniteTree.OperationType.PUT;
{code}

which is responsible for a choosing further tree operation after the closure 
update. And if rows links are equal (which is true for in-place update), the 
{{NOOP}} is chosen and therefore methods {{CacheDataStoreImpl#finishUpdate}} 
and {{GridQueryProcessor#store}} are not called in 
{{CacheDataStoreImpl#invoke}}.

TC tests are OK. Please review.

> SQL: index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>  Labels: iep-1, performance
> Fix For: 2.4
>
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-7015) SQL: index should be updated only when relevant values changed

[Roman Kondakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280556#comment-16280556
 ] 

Roman Kondakov commented on IGNITE-7015:


[~vozerov] please review, tests are OK.

> SQL: index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>  Labels: iep-1, performance
> Fix For: 2.4
>
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-7015) SQL: index should be updated only when relevant values changed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279210#comment-16279210
 ] 

ASF GitHub Bot commented on IGNITE-7015:


GitHub user dolphin1414 opened a pull request:

https://github.com/apache/ignite/pull/3150

IGNITE-7015: Indexes are updated only if the indexed fields are changed.

Now if it's possible to update row in-place (without link changing), and 
indexed columns haven't been changed, no indexes will be updated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7015

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3150


commit d5490b0e54491658e5fddc95e8d9779614e6f4ad
Author: rkondakov 
Date:   2017-12-05T16:49:48Z

IGNITE-7015: Indexes are updated only if the indexed fields are changed.




> SQL: index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>  Labels: iep-1, performance
> Fix For: 2.4
>
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-7015) SQL: index should be updated only when relevant values changed

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276734#comment-16276734
 ] 

Vladimir Ozerov commented on IGNITE-7015:
-

Probably we can re-use existing cleanup infrastructure, see 
{{IgniteH2Indexing.STATEMENT_CACHE_THREAD_USAGE_TIMEOUT}}.

> SQL: index should be updated only when relevant values changed
> --
>
> Key: IGNITE-7015
> URL: https://issues.apache.org/jira/browse/IGNITE-7015
> Project: Ignite
>  Issue Type: Task
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Roman Kondakov
>  Labels: iep-1, performance
> Fix For: 2.4
>
>
> See {{GridH2Table.update}} method. Whenever value is updated, we propagate it 
> to all indexes. Consider the following case:
> 1) Old row is not null, so this is "update", not "create".
> 2) Link hasn't changed
> 3) Indexed fields haven't changed
> If all conditions are met, we can skip index update completely, as state 
> before and after will be the same. This is especially important when 
> persistence is enabled because currently we generate unnecessary dirty pages 
> what increases IO pressure.
> Suggested fix:
> 1) Iterate over index columns, skipping key and affinity columns (as they are 
> guaranteed to be the same);
> 2) Compare relevant index columns of both old and new rows
> 3) If all columns are equal, do nothing.
> Fields should be read through {{GridH2KeyValueRowOnheap#getValue}}, because 
> in this case we will re-use value cache transparently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)