[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630029#comment-16630029 ] Vladimir Ozerov commented on IGNITE-6854: - Moved to 2.8 due to inactivity. Please feel free to move back if you will be able to complete the ticket by AI 2.7 code freeze date, September 30, 2018. > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.3 >Reporter: Mulugeta Mammo >Priority: Major > Fix For: 2.8 > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583544#comment-16583544 ] Maxim Muzafarov commented on IGNITE-6854: - [~mulugetam] This addition sounds very interesting for me and luckily for myself I found this task. Can you initiate discussion on dev.list and share details of your impletentation? So, we will proceed with merging it. You can assign this issue to youself, run TC build and set status as "Patch available", so anyone can join to review. > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.3 >Reporter: Mulugeta Mammo >Priority: Major > Fix For: 2.7 > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548171#comment-16548171 ] Mulugeta Mammo commented on IGNITE-6854: Pull request: https://github.com/apache/ignite/pull/4381 > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.3 >Reporter: Mulugeta Mammo >Priority: Major > Fix For: 2.7 > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533790#comment-16533790 ] Mulugeta Mammo commented on IGNITE-6854: Hi, I have pulled down the patch and will post an enhancement proposal and pull request for a new version. Thanks > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.3 >Reporter: Mulugeta Mammo >Priority: Major > Fix For: 2.7 > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469690#comment-16469690 ] Mulugeta Mammo commented on IGNITE-6854: I have included the patch that optionally enables 3DxPoint persistent store based on the second implementation option outlined in the description. > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.3 >Reporter: Mulugeta Mammo >Priority: Major > Fix For: 2.6 > > Attachments: ignite_3DXPoint.patch > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-6854) Enabling Persistent Memory for Ignite
[ https://issues.apache.org/jira/browse/IGNITE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252130#comment-16252130 ] Denis Magda commented on IGNITE-6854: - [~mulugeta], No doubts, it will be beneficial for Ignite to integrate with the persistent memory. Personally, I truly like option 2. Please initiate a discussion on Apache Ignite dev list bringing the proposed integration to the attention of a wider audience. First, we need to assess the complexity and confirm that's either of options is feasible and worth the efforts of both Ignite and Intel developers. Don't forget to register on the dev list before sending a message there: https://ignite.apache.org/community/resources.html#mail-lists > Enabling Persistent Memory for Ignite > - > > Key: IGNITE-6854 > URL: https://issues.apache.org/jira/browse/IGNITE-6854 > Project: Ignite > Issue Type: New Feature > Security Level: Public(Viewable by anyone) >Affects Versions: 2.3 >Reporter: Mulugeta Mammo > Fix For: 2.4 > > > Ignite, when persistence mode is enabled, stores data and indexes on disk. To > minimize the latency of disks, several tuning options can be applied. Setting > the page size of a memory region to match the page size of the underlying > storage, using a separate disk for the WAL, and using production-level SSDs > are just a few of them [ > https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning > ]. > > A persistent memory store with low latency and high capacity offers a viable > alternative to disks. In light of this, we are proposing to make use of our > Low Level Persistent Library (LLPL), > https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory > storage for Ignite. > > At this point, we envision two distinct implementation options: > # Data and indexes will continue to be stored in the off-heap memory but the > disk will be replaced by a persistent memory. Since persistence memory in > this option is not a file system, the logic currently offered by WAL file and > the partition files would have to be implemented from scratch. > # In this option, we eliminate the current check-point process and the WAL > file. We will use a memory region defined by LLPL to store data and indexes. > There will be no off-heap memory. DRAM will be exclusively used to store hot > cache entries just like the on-heap cache is in the current implementation. > > > In both cases, there are more details and subtleties that have to handled – > e.g. the atomic and transactional guarantees offered. More clarifications > will be given as we go along. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
