[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368553#comment-16368553 ]
Anoop Sam John commented on HBASE-20003: ---------------------------------------- Will come up with the doc soon Andy. The point you raised is valid and we thought. Even today when there is a 3 node replication in HDFS (for WAL file) and all the 3 servers (where the data is replicated) goes down, the wal is not available for the replay and so temp data not available. > WALLess HBase on Persistent Memory > ---------------------------------- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature > Reporter: Anoop Sam John > Assignee: Anoop Sam John > Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > -Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > -Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > -Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > -Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)