[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2019-10-10 Thread Maxim Muzafarov (Jira) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948503#comment-16948503
 ] 

Maxim Muzafarov commented on IGNITE-8873:
-

Moved to the next release due to inactivity. Please, feel free to move it back 
if you will be able to complete the ticket by 2.8 code freeze date, December 2, 
2019.


> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2019-06-25 Thread Alexei Scherbakov (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872682#comment-16872682
 ] 

Alexei Scherbakov commented on IGNITE-8873:
---

[~dmagda]

This method was added to address exactly the case where huge (tens of 
terabytes) cache have to be efficiently fully scanned.
It was already successfully used in production by some Ignite users as far as I 
know.
The main idea behind per partition preloading API same as for other methods 
working with partitions: affinity run/call, scan by partition, SQL query by 
partition(s).
I suggest to keep this method for advanced use cases and add some more "high 
level" APIs like you have proposed.

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-27 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665995#comment-16665995
 ] 

ASF GitHub Bot commented on IGNITE-8873:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5091


> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-26 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665269#comment-16665269
 ] 

ASF GitHub Bot commented on IGNITE-8873:


GitHub user ascherbakoff opened a pull request:

https://github.com/apache/ignite/pull/5091

IGNITE-8873 Test fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8873

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5091


commit 67fa13bf4f5755aca5f43db0819b9cb19a197496
Author: Aleksei Scherbakov 
Date:   2018-09-26T09:00:26Z

IGNITE-8873 wip.

commit 5a9e712161578087f55895072e5d6fde53ab722d
Author: Aleksei Scherbakov 
Date:   2018-09-26T17:13:09Z

IGNITE-8873 wip.

commit 3653b5e7f6687de34a70ae99988df2c8536d421f
Author: ascherbakoff 
Date:   2018-09-26T21:08:28Z

IGNITE-8873 wip.

commit 390ab35492f3b54093025fd3bffc2a3798105fc3
Author: Aleksei Scherbakov 
Date:   2018-10-19T14:00:32Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-8873

commit 7ecdc9d01148ab4b2531e29a81250e6c918428a7
Author: Aleksei Scherbakov 
Date:   2018-10-19T16:16:13Z

IGNITE-8873 wip.

commit 7018264fae040f04f1eefcc144295047f3f20255
Author: Aleksei Scherbakov 
Date:   2018-10-19T16:18:24Z

IGNITE-8873 wip.

commit f3ff3499695643f1e319af64db5deb1198f9d7ad
Author: Aleksei Scherbakov 
Date:   2018-10-19T16:21:50Z

Merge branch 'ignite-8873' of https://github.com/gridgain/apache-ignite 
into ignite-8873

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheAdapter.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheOffheapManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheOffheapManagerImpl.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/local/GridLocalCache.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
#   
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java

commit 9b4180f35bc571484c4d07418dfaf4c83b165c99
Author: Aleksei Scherbakov 
Date:   2018-10-19T16:22:00Z

Merge branch 'ignite-8873' of https://github.com/gridgain/apache-ignite 
into ignite-8873

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheAdapter.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheOffheapManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheOffheapManagerImpl.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/local/GridLocalCache.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
#   
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java

commit 3332500da74a37ca3cda4068683d7dba0c5c8bed
Author: Aleksei Scherbakov 
Date:   2018-10-22T17:01:05Z

IGNITE-8873 wip.

commit 4c4b2050f8f005e101801d65db267ca14fdead6a
Author: Aleksei Scherbakov 
Date:   2018-10-22T17:01:26Z

IGNITE-8873 wip.

commit c594dc8d85681ef88f7a2a73c9e5935ef742648a
Author: Aleksei Scherbakov 
Date:   2018-10-22T17:02:12Z

IGNITE-8873 wip.

commit ee278751a3c1078277159e51fca24b3e167a3aca
Author: Aleksei Scherbakov 
Date:   2018-10-23T09:36:31Z

IGNITE-8873 wip.

commit 5e25e88476adf43b0c2d5e8a2566ede9aef50fa3
Author: Aleksei Scherbakov 
Date:   2018-10-23T09:43:33Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-8873

commit 54e7fe1801f8a20036d6b665a5516aa6cddfa981
Author: Aleksei Scherbakov 
Date:   2018-10-23T09:59:41Z

IGNITE-8873 wip.

commit 5daab6029f0e54ba60162784e1e170f170c05d82
Author: Aleksei Scherbakov 
Date:   2018-10-24T08:56:40Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-8873

commit 68ba4496622bb71eb92811f7fec4e162ac5e94dd
Author: Aleksei Scherbakov 
Date:   2018-10-24T09:13:32Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-8873

commit 538f199d9e82fb1810d57b466072200c3aec7d52
Author: Aleksei Scherbakov 
Date:   2018-10-25T09:07:12Z

IGNITE-8873 wip.

commit 2316ae10490c2910cf9b46abe1ba0feed9008011
Author: Aleksei Scherbakov 
Date:   2018-10-25T15:23:22Z

IGNITE-8873 wip.

commit 06a6f099e0613ceac1ad63c27df60d4afce8ae71
Author: Aleksei Scherbakov 
Date:   2018-10-25T15:46:08Z

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-26 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664994#comment-16664994
 ] 

ASF GitHub Bot commented on IGNITE-8873:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5053


> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-26 Thread Alexey Goncharuk (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664991#comment-16664991
 ] 

Alexey Goncharuk commented on IGNITE-8873:
--

[~ascherbakov], thanks, I merged your changes to master. Please create and link 
a documentation ticket and resolve this ticket.

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-25 Thread Alexei Scherbakov (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663918#comment-16663918
 ] 

Alexei Scherbakov commented on IGNITE-8873:
---

[~agoncharuk],

1) The main use case for this method is preloading for iteration using scan or 
sql query over partition which is almost always done over primary partition. 
Backup partitions are not intended for use from public API, so I do not think 
it's necessary to preload backups.
I've added {{preloadPartitionLocal}} method as you suggested which could be 
used for preloading of any local partition. By combining compute API and this 
method backup partition can be preloaded if needed.
2) Done
3) Done. Added two new tests {{testPreloadPartitionInMemoryRemote, 
testPreloadPartitionInMemoryLocal}}

Please review.


> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-25 Thread Alexey Goncharuk (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663690#comment-16663690
 ] 

Alexey Goncharuk commented on IGNITE-8873:
--

[~ascherbakov], a few comments:
1) From the change I see that the method {{preloadPartition()}} is supposed to 
be cluster-wide, however {{affinityRun}} will send the runnable only to one 
node, instead of primary and backups. Is it how it is supposed to work? I think 
it makes sense to preload both primary and backups. Also, should we add 
{{preloadPartitionLocal()}} method which will not broadcast any jobs but 
preload partition only if it exists locally?
2) Please add a proper javadoc to public API methods so a user understands what 
they do and what drawbacks it may have on page memory and performance
3) I think the methods should throw/return failed future with an 
{{IgniteException}} if the method is called on a non-persistent cache.

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-25 Thread Alexei Scherbakov (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663670#comment-16663670
 ] 

Alexei Scherbakov commented on IGNITE-8873:
---

[~agoncharuk], 

Please review.

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-25 Thread Ignite TC Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663668#comment-16663668
 ] 

Ignite TC Bot commented on IGNITE-8873:
---

{panel:title=No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity Run All 
Results|http://ci.ignite.apache.org/viewLog.html?buildId=2154428buildTypeId=IgniteTests24Java8_RunAll]

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-10-22 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659584#comment-16659584
 ] 

ASF GitHub Bot commented on IGNITE-8873:


GitHub user ascherbakoff opened a pull request:

https://github.com/apache/ignite/pull/5053

IGNITE-8873 Preload cache partition into page memory.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8873

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5053






> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Alexei Scherbakov
>Priority: Major
> Fix For: 2.8
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-8873) Optimize cache scans with enabled persistence.

2018-07-10 Thread Vladislav Pyatkov (JIRA) 


[ 
https://issues.apache.org/jira/browse/IGNITE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538387#comment-16538387
 ] 

Vladislav Pyatkov commented on IGNITE-8873:
---

I think, that will to by done across preload every partition pages before 
iterate over entry.

And other idea: to implement specific configuration parameter (like 
\{{CacheConfiguration#warnUpOnStart(boolean)}}), that preload all pages from 
storage for particular cache (cache group) after the cache started.

Anyone have idea, how to implement it in public API?

> Optimize cache scans with enabled persistence.
> --
>
> Key: IGNITE-8873
> URL: https://issues.apache.org/jira/browse/IGNITE-8873
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Priority: Major
> Fix For: 2.7
>
>
> Currently cache scans with enabled persistence involve link resolution, which 
> can lead to radom disk access resulting in bad performace on SAS disks.
> One possibility is to preload cache data pages to remove slow random disk 
> access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)